Re: Does variation in no of columns in rows over the column family has any performance impact ?

[Peter Schuller]

> Does huge variation in no. of columns in rows, over the column family
> has *any* impact on the performance ?
>
> Can I have like just 100 columns in some rows and like hundred
> thousands of columns in another set of rows, without any downsides ?

If I interpret your question the way I think you mean it, then no,
Cassandra doesn't "do" anything with the data such that the smaller
rows are somehow directly less efficient because there are other rows
that are bigger. It doesn't affect the on-disk format or the on-disk
efficiency of accessing the rows.

However, there are almost always indirect effects when it comes to
performance, in and particular storage systems. In the case of
Cassandra, the *variation* itself should not impose a direct
performance penalty, but there are potential other effects. For
example the row cache is only useful for small works, so if you are
looking to use the row cache the huge rows would perhaps prevent that.
This could be interpreted as a performance impact on the smaller rows
by the larger rows Compaction may become more expensive due to
e.g. additional GC pressure resulting from
large-but-still-within-in-memory-limits rows being compacted (or not,
depending on JVM/GC settings). There is also the effect of cache
locality as data set grows, and the cache locality for the smaller
rows will likely be worse than had they been in e.g. a separate CF.

Those are just three random example; I'm just trying to make the point
that "without any downsides" is a very strong and blanket requirement
for making the decision to mix small rows with larger ones.

-- 
/ Peter Schuller

Re: Using a synchronized counter that keeps track of no of users on the application & using it to allot UserIds/ keys to the new users after sign up

[David Boxenhorn]

Why not synchronize on the client side? Make sure that the process that
allocates user ids runs on only a single machine, in a synchronized method,
and uses QUORUM for its reads and writes to Cassandra?

On Sun, Feb 6, 2011 at 11:02 PM, Aaron Morton wrote:

> If you mix mysql and Cassandra you risk creating a single point of failure
> around the mysql system.
>
> If you have use data that changes infrequently, a row cache in cassandra
> will give you fast reads.
>
> Aaron
>
> On 5/02/2011, at 8:13 AM, Aklin_81  wrote:
>
> > Thanks so much Ryan for the links; I'll definitely take them into
> > consideration.
> >
> > Just another thought which came to my mind:-
> > perhaps it may be beneficial to store(or duplicate) some of the data
> > like the Login credentials & particularly userId to User's Name
> > mapping, etc (which is very heavily read), in a fast MyISAM table.
> > This could solve the problem of keys though auto-generated unique &
> > sequential primary keys. I could use the same keys for Cassandra rows
> > for that user. And also since Cassandra reads are relatively slow, it
> > makes sense to store data like userId to Name mapping in MyISAM as
> > this data would be required after almost all queries to the database.
> >
> > Regards
> > -Asil
> >
> >
> >
> > On Fri, Feb 4, 2011 at 10:14 PM, Ryan King  wrote:
> >> On Thu, Feb 3, 2011 at 9:12 PM, Aklin_81  wrote:
> >>> Thanks Matthew & Ryan,
> >>>
> >>> The main inspiration behind me trying to generate Ids in sequential
> >>> manner is to reduce the size of the userId, since I am using it for
> >>> heavy denormalization. UUIDs are 16 bytes long, but I can also have a
> >>> unique Id in just 4 bytes, and since this is just a one time process
> >>> when the user signs-up, it makes sense to try cutting down the space
> >>> requirements, if it is feasible "without any downsides"(!?).
> >>>
> >>> I am also using userIds to attach to Id of the other data of the user
> >>> on my application. If I could reduce the userId size that I can also
> >>> reduce the size of other Ids, I could drastically cut down the space
> >>> requirements.
> >>>
> >>>
> >>> [Sorry for this question is not directly related to cassandra but I
> >>> think Cassandra factors here because of its  tuneable consistency]
> >>
> >> Don't generate these ids in cassandra. Use something like snowflake,
> >> flickr's ticket servers [2] or zookeeper sequential nodes.
> >>
> >> -ryan
> >>
> >>
> >> 1. http://github.com/twitter/snowflake
> >> 2.
> http://code.flickr.com/blog/2010/02/08/ticket-servers-distributed-unique-primary-keys-on-the-cheap/
> >>
>

OOM during batch_mutate

[Patrik Modesto]

Hi all!

I'm running into OOM problem during batch_mutate. I've a test cluster
of two servers, 32GB and 16GB RAM, real HW. I've one keyspace and one
CF with 1,4mil rows, each 10 columns. A row is around 5k in size. I
run Hadoop MR task that reads one column and generates Mutation that
updates another column, it writes  about 1k string there. Strange
thing is I can run this MR three times without any error on a freshly
created/filled CF but the 4th time Cassandra on the 16GB server hits
OOM. After I run 'nodetool compact' on the 16GB server I can run the
MR again without any OOM one or two times. I'm quite clueless on what
to change in Cassandra/CF settings. Can anyone help me?

Regards,
Patrik

output.log:

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor30]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor34]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor20]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor10]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor32]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor19]
[Unloading class sun.reflect.GeneratedMethodAccessor9]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor33]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor16]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor15]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor23]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor43]
 WARN 09:28:46,172 Dropped 2 MUTATION messages in the last 5000ms
 WARN 09:30:10,555 Dropped 7 REQUEST_RESPONSE messages in the last 5000ms
 INFO 09:30:10,556 Pool NameActive   Pending
 INFO 09:30:10,556 ReadStage 0 0
 INFO 09:30:10,556 RequestResponseStage  050
 INFO 09:30:10,557 ReadRepair0 0
 INFO 09:30:10,557 MutationStage1626
 INFO 09:30:10,557 GossipStage   1 8
 INFO 09:30:10,558 AntiEntropyStage  0 0
 INFO 09:30:10,558 MigrationStage0 0
 INFO 09:30:10,558 StreamStage   0 0
ERROR 09:30:10,561 Internal error processing batch_mutate
java.lang.NoClassDefFoundError: Could not initialize class
org.apache.cassandra.thrift.TimedOutException
at 
org.apache.cassandra.thrift.CassandraServer.doInsert(CassandraServer.java:417)
at 
org.apache.cassandra.thrift.CassandraServer.batch_mutate(CassandraServer.java:385)
at 
org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.process(Cassandra.java:3036)
at 
org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2555)
at 
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:167)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
ERROR 09:30:10,675 Fatal exception in thread Thread[MutationStage:14,5,main]
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2786)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at 
org.apache.cassandra.utils.FBUtilities.writeByteArray(FBUtilities.java:266)
at 
org.apache.cassandra.db.ColumnSerializer.serialize(ColumnSerializer.java:56)
at 
org.apache.cassandra.db.ColumnSerializer.serialize(ColumnSerializer.java:35)
at 
org.apache.cassandra.db.ColumnFamilySerializer.serializeForSSTable(ColumnFamilySerializer.java:87)
at 
org.apache.cassandra.db.ColumnFamilySerializer.serialize(ColumnFamilySerializer.java:73)
at 
org.apache.cassandra.db.RowMutationSerializer.freezeTheMaps(RowMutation.java:364)
at 
org.apache.cassandra.db.RowMutationSerializer.serialize(RowMutation.java:375)
at 
org.apache.cassandra.db.RowMutationSerializer.serialize(RowMutation.java:353)
at 
org.apache.cassandra.db.RowMutation.getSerializedBuffer(RowMutation.java:274)
at org.apache.cassandra.db.RowMutation.apply(RowMutation.java:194)
at 
org.apache.cassandra.service.StorageProxy$1.runMayThrow(StorageProxy.java:202)
at 
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
ERROR 09:30:10,675 Internal error processing batch_mutate
java.lang.NoClassDefFoundError: Could not initialize class
org.apache.cassandra.thrift.T

Re: Does variation in no of columns in rows over the column family has any performance impact ?

[Aditya Narayan]

Thanks for the detailed explanation Peter! Definitely cleared my doubts !



On Mon, Feb 7, 2011 at 1:52 PM, Peter Schuller
 wrote:
>> Does huge variation in no. of columns in rows, over the column family
>> has *any* impact on the performance ?
>>
>> Can I have like just 100 columns in some rows and like hundred
>> thousands of columns in another set of rows, without any downsides ?
>
> If I interpret your question the way I think you mean it, then no,
> Cassandra doesn't "do" anything with the data such that the smaller
> rows are somehow directly less efficient because there are other rows
> that are bigger. It doesn't affect the on-disk format or the on-disk
> efficiency of accessing the rows.
>
> However, there are almost always indirect effects when it comes to
> performance, in and particular storage systems. In the case of
> Cassandra, the *variation* itself should not impose a direct
> performance penalty, but there are potential other effects. For
> example the row cache is only useful for small works, so if you are
> looking to use the row cache the huge rows would perhaps prevent that.
> This could be interpreted as a performance impact on the smaller rows
> by the larger rows Compaction may become more expensive due to
> e.g. additional GC pressure resulting from
> large-but-still-within-in-memory-limits rows being compacted (or not,
> depending on JVM/GC settings). There is also the effect of cache
> locality as data set grows, and the cache locality for the smaller
> rows will likely be worse than had they been in e.g. a separate CF.
>
> Those are just three random example; I'm just trying to make the point
> that "without any downsides" is a very strong and blanket requirement
> for making the decision to mix small rows with larger ones.
>
> --
> / Peter Schuller
>

Java Cassandra Driver: Using CQL

[Vivek Mishra]

Hi,
Recently I worked on implementation of java jdbc driver for cassandra using CQL.
Given below is an example code base(with basic features)  about how to use it:

import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;



public class Cass
{
public static void main(String[] args) {
  java.sql.Connection con =null;
try {
 String selectQ = "SELECT \"first\", 
\"last\" FROM Users WHERE KEY=\"jsmith\"";

Class.forName("com.impetus.jdbc.cassandra.api.CassandraDriver");
// con = 
DriverManager.getConnection("jdbc:cassandra:localhost@9160");
con = 
DriverManager.getConnection("jdbc:cassandra:root/root@localhost:9160/root1");

 //With Statement
 scrollResultset(withStatement(con));

 //With PreparedStatement.
 scrollResultset(withPreparedStatement(con, selectQ));

 //Update/INSERT
 withUpdateStatement(con);
 String updateSelect =  "SELECT \"firstN\", \"lastN\" FROM Users 
WHERE KEY=\"jsmith\"";
 scrollResultset(withPreparedStatement(con,updateSelect));

 //Delete
 withDeleteStatement(con);
 scrollResultset(withPreparedStatement(con,updateSelect));

} catch (ClassNotFoundException e) {
e.printStackTrace();
}
catch (SQLException e) {
e.printStackTrace();
} finally{
if(con !=null) {
try {

con.close();
} catch 
(SQLException e) {

// TODO Auto-generated catch block

e.printStackTrace();
}
con = null;
}
}
}

/**
*
 * @param con
* @return
* @throws SQLException
*/
private static ResultSet withStatement(Connection con) throws 
SQLException{
//String useQ = "USE Keyspace1";
 String selectQ = "SELECT \"first\", \"last\" FROM Users 
WHERE KEY=\"jsmith\"";
   Statement stmt = con.createStatement();
//   stmt.execute(useQ);
   return stmt.executeQuery(selectQ);
}

/**
*
 * @param con
* @return
* @throws SQLException
*/
private static ResultSet withPreparedStatement(Connection con, 
String selectQ) throws SQLException{
//String useQ = "USE Keyspace1";
// PreparedStatement statement = con.prepareStatement(useQ);
// statement.execute();
PreparedStatement statement = 
con.prepareStatement(selectQ);
 return statement.executeQuery();

}

/**
*
 * @param rSet
* @throws SQLException
*/
private static void scrollResultset(ResultSet rSet) throws 
SQLException {

while(rSet.next()){
 System.out.println(rSet.getString(0));
 System.out.println(rSet.getString("last"));
 System.out.println(rSet.getString("lastN"));

 }
}

private static void withUpdateStatement(Connection con) throws 
SQLException {
String updateQ =  "UPDATE Users SET \"firstN\" 
= \"vivekn\", \"lastN\" = \"mishran\" WHERE KEY = \"jsmith\"";
PreparedStatement statement = 
con.prepareStatement(updateQ);
statement.execute();
}

private static void withDeleteStatement(Connection con) throws 
SQLException {
String deleteQ = "DELETE \"firstN\", \"lastN\" 
FROM Users WHERE KEY=\"jsmith\"";
PreparedStatement statement = 
con.prepareStatement(deleteQ);
statement.execute();
}
}


I am not sure if there is any JIRA 

[0.7.1] Error in ThreadPoolExecutor

[Patrik Modesto]

Hi,

on my two-node test setup I get repeatedly following error:

The 10.0.18.129 server log:

 INFO 14:10:37,707 Node /10.0.18.99 has restarted, now UP again
 INFO 14:10:37,708 Checking remote schema before delivering hints
 INFO 14:10:37,708 Sleeping 45506ms to stagger hint delivery
 INFO 14:10:37,709 Node /10.0.18.99 state jump to normal
 INFO 14:11:23,215 Started hinted handoff for endpoint /10.0.18.99
ERROR 14:11:23,884 Error in ThreadPoolExecutor
java.lang.RuntimeException: java.lang.IllegalArgumentException
at 
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.IllegalArgumentException
at java.nio.Buffer.position(Buffer.java:218)
at 
org.apache.cassandra.utils.ByteBufferUtil.string(ByteBufferUtil.java:117)
at 
org.apache.cassandra.utils.ByteBufferUtil.string(ByteBufferUtil.java:111)
at 
org.apache.cassandra.db.HintedHandOffManager.getTableAndCFNames(HintedHandOffManager.java:237)
at 
org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:306)
at 
org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:88)
at 
org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:385)
at 
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
... 3 more
ERROR 14:11:23,885 Fatal exception in thread Thread[HintedHandoff:1,1,main]
java.lang.RuntimeException: java.lang.IllegalArgumentException
at 
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.IllegalArgumentException
at java.nio.Buffer.position(Buffer.java:218)
at 
org.apache.cassandra.utils.ByteBufferUtil.string(ByteBufferUtil.java:117)
at 
org.apache.cassandra.utils.ByteBufferUtil.string(ByteBufferUtil.java:111)
at 
org.apache.cassandra.db.HintedHandOffManager.getTableAndCFNames(HintedHandOffManager.java:237)
at 
org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:306)
at 
org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:88)
at 
org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:385)
at 
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
... 3 more


The 10.0.18.99 server log:

 INFO 14:10:37,691 Binding thrift service to /0.0.0.0:9160
 INFO 14:10:37,693 Using TFastFramedTransport with a max frame size of
15728640 bytes.
 INFO 14:10:37,695 Listening for thrift clients...
 INFO 14:10:38,337 GC for ParNew: 954 ms, 658827608 reclaimed leaving
966732432 used; max is 4265607168
 INFO 14:11:27,142 Started hinted handoff for endpoint /10.0.18.129
ERROR 14:11:27,370 Error in ThreadPoolExecutor
java.lang.RuntimeException: java.lang.IllegalArgumentException
at 
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.IllegalArgumentException
at java.nio.Buffer.position(Buffer.java:218)
at 
org.apache.cassandra.utils.ByteBufferUtil.string(ByteBufferUtil.java:117)
at 
org.apache.cassandra.utils.ByteBufferUtil.string(ByteBufferUtil.java:111)
at 
org.apache.cassandra.db.HintedHandOffManager.getTableAndCFNames(HintedHandOffManager.java:237)
at 
org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:306)
at 
org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:88)
at 
org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:385)
at 
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
... 3 more
ERROR 14:11:27,371 Fatal exception in thread Thread[HintedHandoff:1,1,main]
java.lang.RuntimeException: java.lang.IllegalArgumentException
at 
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.IllegalArgu

Re: OOM during batch_mutate

[Patrik Modesto]

I forgot to mention I use 0.7.0 stable version.

HTH,
Patrik

Re: OOM during batch_mutate

[Patrik Modesto]

Just tried current 0.7.1 from cassandra-0.7 branch and it does the
same. OOM after three runs.

-Xm* setting is computed by cassandra-env.sh like this:  -Xms8022M
-Xmx8022M -Xmn2005M

What am I doing wrong?

Thanks,
Patrik

On Mon, Feb 7, 2011 at 14:18, Patrik Modesto  wrote:
> I forgot to mention I use 0.7.0 stable version.
>
> HTH,
> Patrik
>

[0.7.1] more exceptions: Illegal mode

[Patrik Modesto]

INFO 15:30:49,647 Compacted to
/www/foo/cassandra/data/foo/Url-tmp-f-767-Data.db.  4,199,999,762 to
4,162,579,242 (~99% of original) bytes for 379,179 keys.  Time:
137,149ms.
ERROR 15:30:49,699 Fatal exception in thread Thread[CompactionExecutor:1,1,main]
java.lang.RuntimeException: java.lang.IllegalArgumentException:
Illegal mode "w" must be one of "r", "rw", "rws", or "rwd"
at 
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.IllegalArgumentException: Illegal mode "w" must
be one of "r", "rw", "rws", or "rwd"
at java.io.RandomAccessFile.(RandomAccessFile.java:197)
at 
org.apache.cassandra.io.util.BufferedRandomAccessFile.(BufferedRandomAccessFile.java:116)
at 
org.apache.cassandra.io.sstable.CacheWriter.saveCache(CacheWriter.java:48)
at 
org.apache.cassandra.db.CompactionManager$9.runMayThrow(CompactionManager.java:746)
at 
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
... 6 more
ERROR 15:30:49,699 Fatal exception in thread Thread[CompactionExecutor:1,1,main]
java.lang.RuntimeException: java.lang.IllegalArgumentException:
Illegal mode "w" must be one of "r", "rw", "rws", or "rwd"
at 
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.IllegalArgumentException: Illegal mode "w" must
be one of "r", "rw", "rws", or "rwd"
at java.io.RandomAccessFile.(RandomAccessFile.java:197)
at 
org.apache.cassandra.io.util.BufferedRandomAccessFile.(BufferedRandomAccessFile.java:116)
at 
org.apache.cassandra.io.sstable.CacheWriter.saveCache(CacheWriter.java:48)
at 
org.apache.cassandra.db.CompactionManager$9.runMayThrow(CompactionManager.java:746)
at 
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
... 6 more

Re: [0.7.1] more exceptions: Illegal mode

[Thibaut Britz]

I think this is related to a faulty disk.


On Mon, Feb 7, 2011 at 3:35 PM, Patrik Modesto  wrote:
> INFO 15:30:49,647 Compacted to
> /www/foo/cassandra/data/foo/Url-tmp-f-767-Data.db.  4,199,999,762 to
> 4,162,579,242 (~99% of original) bytes for 379,179 keys.  Time:
> 137,149ms.
> ERROR 15:30:49,699 Fatal exception in thread 
> Thread[CompactionExecutor:1,1,main]
> java.lang.RuntimeException: java.lang.IllegalArgumentException:
> Illegal mode "w" must be one of "r", "rw", "rws", or "rwd"
>        at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>        at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>        at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.IllegalArgumentException: Illegal mode "w" must
> be one of "r", "rw", "rws", or "rwd"
>        at java.io.RandomAccessFile.(RandomAccessFile.java:197)
>        at 
> org.apache.cassandra.io.util.BufferedRandomAccessFile.(BufferedRandomAccessFile.java:116)
>        at 
> org.apache.cassandra.io.sstable.CacheWriter.saveCache(CacheWriter.java:48)
>        at 
> org.apache.cassandra.db.CompactionManager$9.runMayThrow(CompactionManager.java:746)
>        at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>        ... 6 more
> ERROR 15:30:49,699 Fatal exception in thread 
> Thread[CompactionExecutor:1,1,main]
> java.lang.RuntimeException: java.lang.IllegalArgumentException:
> Illegal mode "w" must be one of "r", "rw", "rws", or "rwd"
>        at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>        at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>        at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.IllegalArgumentException: Illegal mode "w" must
> be one of "r", "rw", "rws", or "rwd"
>        at java.io.RandomAccessFile.(RandomAccessFile.java:197)
>        at 
> org.apache.cassandra.io.util.BufferedRandomAccessFile.(BufferedRandomAccessFile.java:116)
>        at 
> org.apache.cassandra.io.sstable.CacheWriter.saveCache(CacheWriter.java:48)
>        at 
> org.apache.cassandra.db.CompactionManager$9.runMayThrow(CompactionManager.java:746)
>        at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>        ... 6 more
>

Re: OOM during batch_mutate

[sridhar basam]

Looks like you don't have a big enough working set from your GC logs, there
doesn't seem to be a lot being reclaimed in the GC process. The process is
reclaiming a few hundred MB and is running every few seconds. How big are
your caches? The probable reason that it works the first couple times when
you create it due to nothing being in cache as it gets built up.

 Sridhar



On Mon, Feb 7, 2011 at 8:31 AM, Patrik Modesto wrote:

> Just tried current 0.7.1 from cassandra-0.7 branch and it does the
> same. OOM after three runs.
>
> -Xm* setting is computed by cassandra-env.sh like this:  -Xms8022M
> -Xmx8022M -Xmn2005M
>
> What am I doing wrong?
>
> Thanks,
> Patrik
>
> On Mon, Feb 7, 2011 at 14:18, Patrik Modesto 
> wrote:
> > I forgot to mention I use 0.7.0 stable version.
> >
> > HTH,
> > Patrik
> >
>

Re: Does variation in no of columns in rows over the column family has any performance impact ?

[Edward Capriolo]

On Mon, Feb 7, 2011 at 5:40 AM, Aditya Narayan  wrote:
> Thanks for the detailed explanation Peter! Definitely cleared my doubts !
>
>
>
> On Mon, Feb 7, 2011 at 1:52 PM, Peter Schuller
>  wrote:
>>> Does huge variation in no. of columns in rows, over the column family
>>> has *any* impact on the performance ?
>>>
>>> Can I have like just 100 columns in some rows and like hundred
>>> thousands of columns in another set of rows, without any downsides ?
>>
>> If I interpret your question the way I think you mean it, then no,
>> Cassandra doesn't "do" anything with the data such that the smaller
>> rows are somehow directly less efficient because there are other rows
>> that are bigger. It doesn't affect the on-disk format or the on-disk
>> efficiency of accessing the rows.
>>
>> However, there are almost always indirect effects when it comes to
>> performance, in and particular storage systems. In the case of
>> Cassandra, the *variation* itself should not impose a direct
>> performance penalty, but there are potential other effects. For
>> example the row cache is only useful for small works, so if you are
>> looking to use the row cache the huge rows would perhaps prevent that.
>> This could be interpreted as a performance impact on the smaller rows
>> by the larger rows Compaction may become more expensive due to
>> e.g. additional GC pressure resulting from
>> large-but-still-within-in-memory-limits rows being compacted (or not,
>> depending on JVM/GC settings). There is also the effect of cache
>> locality as data set grows, and the cache locality for the smaller
>> rows will likely be worse than had they been in e.g. a separate CF.
>>
>> Those are just three random example; I'm just trying to make the point
>> that "without any downsides" is a very strong and blanket requirement
>> for making the decision to mix small rows with larger ones.
>>
>> --
>> / Peter Schuller
>>
>

The performance could be variable if you are using operations such as
a get_slice with a large Slice Predicate, large rows take longer to be
de serialized and transferred then smaller rows. I have never
benchmarked this but it would probably take a significant difference
in row size before the size of a row had a noticeable impact.

Re: OOM during batch_mutate

[Patrik Modesto]

On Mon, Feb 7, 2011 at 15:44, sridhar basam  wrote:
> Looks like you don't have a big enough working set from your GC logs, there
> doesn't seem to be a lot being reclaimed in the GC process. The process is
> reclaiming a few hundred MB and is running every few seconds. How big are
> your caches? The probable reason that it works the first couple times when
> you create it due to nothing being in cache as it gets built up.

Hi Sridhar,

thanks for your answer. I didn't set the cache to any specific value,
I use the defaults:

[default@Foo] show keyspaces;
Keyspace: Foo:
  Replication Strategy: org.apache.cassandra.locator.SimpleStrategy
Replication Factor: 1
  Column Families:
ColumnFamily: Url
  Columns sorted by: org.apache.cassandra.db.marshal.UTF8Type
  Row cache size / save period: 0.0/0
  Key cache size / save period: 20.0/3600
  Memtable thresholds: 4.6640625/995/60
  GC grace seconds: 864000
  Compaction min/max thresholds: 4/32
  Read repair chance: 1.0
  Built indexes: []

My row-key is 16bytes long string, so I don't expect the cache would
be a problem.

Patrik

Re: [0.7.1] more exceptions: Illegal mode

[Patrik Modesto]

On Mon, Feb 7, 2011 at 15:42, Thibaut Britz
 wrote:
> I think this is related to a faulty disk.

I'm not sure thats the problem. Cassandra 0.7.0 didn't report any
problem. It started with Cassandra 0.7.1.

Patrik

Re: [0.7.1] more exceptions: Illegal mode

[Jake Luciani]

This sounds like a possible bug since the BRAF was re-written in 0.7.1.
Could you open a ticket?

On Mon, Feb 7, 2011 at 10:32 AM, Patrik Modesto wrote:

> On Mon, Feb 7, 2011 at 15:42, Thibaut Britz
>  wrote:
> > I think this is related to a faulty disk.
>
> I'm not sure thats the problem. Cassandra 0.7.0 didn't report any
> problem. It started with Cassandra 0.7.1.
>
> Patrik
>

Re: CF Read and Write Latency Histograms

[Chris Burroughs]

On 02/04/2011 12:43 PM, Jonathan Ellis wrote:
> Can you create a ticket?


I noticed the same thing. CASSANDRA-2123 created.

Re: Java Cassandra Driver: Using CQL

[Eric Evans]

On Mon, 2011-02-07 at 12:08 +, Vivek Mishra wrote:
> Recently I worked on implementation of java jdbc driver for cassandra
> using CQL. Given below is an example code base(with basic features)
> about how to use it:

[ ... ]

Nice!

> I am not sure if there is any JIRA related to this. With such basic
> things I have this in place.(A lot of work needs to be done on this.)

There is CASSANDRA-1710, which saw some discussion about a possible
JDBC-(like|compliant) driver, but it probably makes sense to create one
specifically for this.  Thus, I give you:
https://issues.apache.org/jira/browse/CASSANDRA-2124

> Please let me know, if I can add this project in relation with any
> Cassandra JIRA, else planning to open source it on google code. 

Absolutely, CASSANDRA-2124 is all yours! :)

-- 
Eric Evans
eev...@rackspace.com

Re: Ruby thrift is trying to write Time as string

[Ryan King]

On Sat, Feb 5, 2011 at 10:12 PM, Joshua Partogi  wrote:
> Hi,
>
> I don't know whether my assumption is right or not. When I tried to insert a
> Time value into a column I am getting this exception:
>
> vendor/ruby/1.8/gems/thrift-0.5.0/lib/thrift/protocol/binary_protocol.rb:106:in
> `write_string'
> vendor/ruby/1.8/gems/thrift-0.5.0/lib/thrift/client.rb:35:in `write'
> vendor/ruby/1.8/gems/thrift-0.5.0/lib/thrift/client.rb:35:in `send_message'
> vendor/ruby/1.8/gems/cassandra-0.9.0/lib/./vendor/0.7/gen-rb/cassandra.rb:213:in
> `send_batch_mutate'
> vendor/ruby/1.8/gems/cassandra-0.9.0/lib/./vendor/0.7/gen-rb/cassandra.rb:208:in
> `batch_mutate'
> vendor/ruby/1.8/gems/thrift_client-0.6.0/lib/thrift_client/abstract_thrift_client.rb:115:in
> `send'
> vendor/ruby/1.8/gems/thrift_client-0.6.0/lib/thrift_client/abstract_thrift_client.rb:115:in
> `handled_proxy'
> vendor/ruby/1.8/gems/thrift_client-0.6.0/lib/thrift_client/abstract_thrift_client.rb:57:in
> `batch_mutate'
> vendor/ruby/1.8/gems/cassandra-0.9.0/lib/cassandra/0.7/protocol.rb:8:in
> `_mutate'
> vendor/ruby/1.8/gems/cassandra-0.9.0/lib/cassandra/cassandra.rb:130:in
> `insert'
>
> But I am not getting any error if I insert a Time value into a sub-column.
>
> Is this an error or does it suppose to work that way?
>
> Thanks heaps for the insight.

IIRC when we wrote that code the idea was to require strings for
everything and have users do their own serialization. In that context,
the supercolmn behavior is the wrong one here.

-ryan

-- 
@rk

Re: postgis > cassandra?

[Mike Malone]

It's not really the storage of spatial data that's tricky. We use geojson as
a wire-line format at the higher levels of our system (e.g., the HTTP
API). But the hard part is organizing the data for efficient retrieval and
keeping those indices consistent with the data being indexed. Efficient
multi-dimensional indexing is tricky, but that's what you'll need if you
want to support generic spatial querying (overlaps, contains, interacts,
nearest neighbor, etc).

On Sun, Feb 6, 2011 at 1:14 PM, Aaron Morton wrote:

> Here is a recent presentation from simplegeo.com that may provide some
> inspiration
>
> http://strangeloop2010.com/system/talks/presentations/000/014/495/Malone-DimensionalDataDHT.pdf
>
> Can you provide some more details on the data you want to store and queries
> you want to run ?
>
> Aaron
>
> On 6/02/2011, at 7:04 AM, Sean Ochoa  wrote:
>
> That's a good question, Bill.
>
> The data that I'm trying to store begins as a simple point.  But, moving
> fo=
> rward, it will become more like complex geometries.  I assume that I can
> si=
> mply create a JSON-like object and insert it.  Which, for now, that works.
> =
>  I'm just wondering if theres a typical / publicly accepted standard of
> sto=
> ring somewhat complex spatial data in Cassandra.
>
> Additionally, I would like to figure out how one goes about slicing on
> large spatial data sets given situations where, for instance, I would like
> to get all the points in a column-family where the point is within a shape.
>  I guess it boils down to using a spatial comparator of some sort, but I
> haven't seen one, yet.
>
>  - Sean
>
> On Sat, Feb 5, 2011 at 9:51 AM, William R Speirs < 
> bill.spe...@gmail.com> wrote:
>
>> I know nothing about postgis and little about spacial data, but if you're
>> simply talking about data that relates to some latitude & longitude pair,
>> you could have your row key simply be the concatenation of the two:
>> lat:long.
>>
>> Can you provide more details about the type of data you're looking to
>> store?
>>
>> Thanks...
>>
>> Bill-
>>
>>
>> On 02/05/2011 12:22 PM, Sean Ochoa wrote:
>>
>>> Can someone tell me how to represent spatial data (coming from postgis)
>>> in
>>> Cassandra?
>>>
>>>  - Sean
>>>
>>
>
>
> --
> Sean | M (206) 962-7954 | GV (760) 624-8718
>
>

Re: Does variation in no of columns in rows over the column family has any performance impact ?

[Daniel Doubleday]

It depends a little on your write pattern:

- Wide rows tend to get distributed over more sstables so more disk reads are 
necessary. This will become noticeable when you have high io load and reads 
actually hit the discs.
- If you delete a lot slice query performance might suffer: extreme example: 
create 2M cols, delete the first 1M and then ask for the first 10.


On Feb 7, 2011, at 7:07 AM, Aditya Narayan wrote:

> Does huge variation in no. of columns in rows, over the column family
> has *any* impact on the performance ?
> 
> Can I have like just 100 columns in some rows and like hundred
> thousands of columns in another set of rows, without any downsides ?

Best way to detect/fix bitrot today?

[Anand Somani]

Hi,

Our application space is such that there is data that might not be read for
a long time. The data is mostly immutable. How should I approach
detecting/solving the bitrot problem? One approach is read data and let read
repair do the detection, but given the size of data, that does not look very
efficient.

Has anybody solved/workaround this or has any other suggestions to detect
and fix bitrot?


Thanks
Anand

RE: hadoop library conflict with cassandra (hector) 0.7

[Curt Allred]

Hello,
I'm trying to access a Cassandra 0.7 cluster in a hadoop map-reduce job (hadoop 
0.20.2) and seeing a thrift library conflict.  Hadoop uses an older version of 
thrift than hector 0.7, and this older version is getting picked up by my job, 
causing the following exception:

FATAL org.apache.hadoop.mapred.TaskTracker: Error running child :
java.lang.NoSuchMethodError: 
org.apache.thrift.protocol.TProtocol.writeBinary(Ljava/nio/ByteBuffer;)V

The ByteBuffer overload of that method is not in the older thrift library.

I have verified that the correct version of thrift (libthrift-0.5.0.jar) is in 
my map-reduce job's jar.  However its using the older version in the hadoop 
library.

Here are a couple of hadoop tickets on the issue:
https://issues.apache.org/jira/browse/MAPREDUCE-1700
https://issues.apache.org/jira/browse/MAPREDUCE-1938

Anyone find a way around this? (besides modifying the hadoop cluster library 
which is not an option for me).
Thanks,
-Curt

Re: Best way to detect/fix bitrot today?

[Peter Schuller]

> Our application space is such that there is data that might not be read for
> a long time. The data is mostly immutable. How should I approach
> detecting/solving the bitrot problem? One approach is read data and let read
> repair do the detection, but given the size of data, that does not look very
> efficient.

Note that read-repair is not really intended to repair arbitrary
corruptions. Unless I'm mistaken, arbitrary corruption, unless it
triggers a serialization failure that causes row skipping, it's a
toss-up which version of the data is retained (or both, if the
corruption is in the key). Given the same key and column timestamp,
the tie breaker is the volumn value. So depending on whether
corruption results in a "lesser" or "greater" value, you might get the
corrupt or non-corrupt data.

> Has anybody solved/workaround this or has any other suggestions to detect
> and fix bitrot?

My feel/tentative opinion is that the clean fix is for Cassandra to
support strong checksumming at the sstable level.

Deploying on e.g. ZFS would help a lot with this, but that's a problem
for deployment on Linux (which is the recommended platform for
Cassandra).

-- 
/ Peter Schuller

Re: Argh: Data Corruption (LOST DATA) (0.7.0)

[Ben Coverston]

Dan,

Do you have any more information on this issue? Have you been able to 
discover anything from exporing your SSTables to JSON?


Thanks,
Ben

On 1/29/11 12:45 PM, Dan Hendry wrote:


I am once again having severe problems with my Cassandra cluster. This 
time, I straight up cannot read sections of data (consistency level 
ONE). Client side, I am seeing timeout exceptions. On the Cassandra 
node, I am seeing errors as shown below. I don't understand what has 
happened or how to fix it. I also don't understand how I am seeing 
errors on only one node, using consistency level ONE with a rf=2 and 
yet clients are failing. I have tried turning on debug logging but 
that been no help, the logs roll over (20 mb) in < 10 seconds (the 
cluster is being used quite heavily).


My cluster has been working fine for weeks the suddenly, I had a 
corrupt SSTable which caused me all sorts of grief (outlined in 
pervious emails). I was able to solve the problem by turning down the 
max compaction threshold then figuring out which SSTable was corrupt 
by watching which minor compactions failed. After that, I straight up 
deleted the on-disk data. Now I am having problems on a different node 
(but adjacent in the ring) for what I am almost certain is the same 
column family (presumably the same row/column). At this point, the 
data is effectively lost as I know 1 of the 2 replicas was completely 
deleted.


Is there any advice going forward? My next course of action was going 
to be exporting all of the sstables to JSON using the provided tool 
and trying to look it over manually to see what the problem actually 
is (if exporting will even work). I am not sure how useful this will 
be as there is nearly 80 GB of data for this CF on a single node. What 
is more concerning is that I have no idea how this problem initially 
popped up. I have performed hardware tests and nothing seems to be 
malfunctioning. Furthermore, the fact that these issues have 'jumped' 
nodes is a strong indication to me this is a Cassandra problem.


There is a Cassandra bug here somewhere, if only in the way corrupt 
columns are dealt with.


db (85098417 bytes)

ERROR [ReadStage:221] 2011-01-29 12:42:39,153 
DebuggableThreadPoolExecutor.java (line 103) Error in ThreadPoolExecutor


java.lang.RuntimeException: java.io.IOException: Invalid 
localDeleteTime read: -1516572672


at 
org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:124)


at 
org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:47)


at 
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)


at 
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)


at 
org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:108)


at 
org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:283)


at 
org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)


at 
org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)


at 
org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)


at 
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)


at 
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)


at 
org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:118)


at 
org.apache.cassandra.db.filter.QueryFilter.collectCollatedColumns(QueryFilter.java:142)


at 
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1230)


at 
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1107)


at 
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1077)


at org.apache.cassandra.db.Table.getRow(Table.java:384)

at 
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:63)


at 
org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:68)


at 
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:63)


at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)


at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)


at java.lang.Thread.run(Thread.java:662)

Caused by: java.io.IOException: Invalid localDeleteTime read: -1516572672

at 
org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:356)


at 
org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:313)


at 
org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.getNextBlock(IndexedSliceReader.

Java bombs during compaction, please help

[buddhasystem]

Hello,
one node in my 3-machine cluster cannot perform compaction. I tried multiple
times, it ran out of heap space once and I increased it. Now I'm getting the
dump below (after it does run for a few minutes). I hope somebody can shed a
little light on what' going on, because I'm at a loss and this is a real
show stopper.


[me@mymachine]~/cassandra-test% Error occured while compacting keyspace
Tracer
java.util.concurrent.ExecutionException: java.lang.NullPointerException
at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source)
at java.util.concurrent.FutureTask.get(Unknown Source)
at
org.apache.cassandra.db.CompactionManager.performMajor(CompactionManager.java:186)
at
org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1766)
at
org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1236)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source)
at com.sun.jmx.mbeanserver.PerInterface.invoke(Unknown Source)
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown Source)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(Unknown
Source)
at javax.management.remote.rmi.RMIConnectionImpl.access$200(Unknown
Source)
at
javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(Unknown
Source)
at
javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(Unknown
Source)
at javax.management.remote.rmi.RMIConnectionImpl.invoke(Unknown
Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
at sun.rmi.transport.Transport$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown
Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.NullPointerException
at
org.apache.cassandra.io.util.ColumnIterator$1.getKey(ColumnSortedMap.java:276)
at
org.apache.cassandra.io.util.ColumnIterator$1.getKey(ColumnSortedMap.java:263)
at
java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(Unknown Source)
at java.util.concurrent.ConcurrentSkipListMap.(Unknown Source)
at
org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:384)
at
org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:332)
at
org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:137)
at
org.apache.cassandra.io.PrecompactedRow.(PrecompactedRow.java:78)
at
org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:139)
at
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:108)
at
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:43)
at
org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
at
org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
at
org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
at
org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:427)
at
org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.j

Re: Best way to detect/fix bitrot today?

[Anthony John]

Some RAID storage might do it, potentially more efficiently!!

Rhetorical question - Does Cassandra's architecture of reconciling reads
over multiple copies of the same data provide an even more interesting
answer? I submit - YES!

All bitrot protection mechanisms involve some element of redundant storage -
to verify and reconstruct any rot. Cassandra can do this on JBODs with the
appropriate Replication Factor (say > 3). Granted that the total storage in
terms of number of disks might exceed the other alternatives, but at the
lowest tier, using JBODs, the cost might actually be lesser.

Food for thought, or wild imagination ?

-JA

On Mon, Feb 7, 2011 at 2:09 PM, Peter Schuller
wrote:

> > Our application space is such that there is data that might not be read
> for
> > a long time. The data is mostly immutable. How should I approach
> > detecting/solving the bitrot problem? One approach is read data and let
> read
> > repair do the detection, but given the size of data, that does not look
> very
> > efficient.
>
> Note that read-repair is not really intended to repair arbitrary
> corruptions. Unless I'm mistaken, arbitrary corruption, unless it
> triggers a serialization failure that causes row skipping, it's a
> toss-up which version of the data is retained (or both, if the
> corruption is in the key). Given the same key and column timestamp,
> the tie breaker is the volumn value. So depending on whether
> corruption results in a "lesser" or "greater" value, you might get the
> corrupt or non-corrupt data.
>
> > Has anybody solved/workaround this or has any other suggestions to detect
> > and fix bitrot?
>
> My feel/tentative opinion is that the clean fix is for Cassandra to
> support strong checksumming at the sstable level.
>
> Deploying on e.g. ZFS would help a lot with this, but that's a problem
> for deployment on Linux (which is the recommended platform for
> Cassandra).
>
> --
> / Peter Schuller
>

time to live rows

[Kallin Nagelberg]

Hey,

I have read about the new TTL columns in Cassandra 0.7. In my case I'd
like to expire an entire row automatically after a certain amount of
time. Is this possible as well?

Thanks,
-Kal

Deleted columns still coming back; CASSANDRA-{1748,1837} alive in 0.6.x?

[Scott McCarty]

Hi,

Does anyone know if anything similar to
https://issues.apache.org/jira/browse/CASSANDRA-1748 or
https://issues.apache.org/jira/browse/CASSANDRA-1837 exists in 0.6.x
releases?  Both of those bugs look like they were introduced, found, and
fixed in 0.7, and CASSANDRA-1837 comments indicate that 0.6 tests passed but
I wanted to double check on this because we've been seeing deleted columns
reappear in our cluster.

Or, maybe if someone has an idea of what might be behind the following
oddness:  columns are deleted, then about two days later (which is what we
have GCGraceSeconds set to) they pop back up again, shortly after running a
"nodetool repair" on all the nodes in the cluster.

One tell-tale clue is that the timestamps on the columns that the client
logs show were deleted look to be the original timestamps, as they are far
back in the past and our code doesn't create timestamp values in the past.
 A bunch of investigation work has us almost convinced that the deleted
columns are popping up in large numbers after a repair is done.

I've posted a message to this list before regarding deleted columns coming
back and one suggestion received was to assume it's a client-side bug, so
we've done a ton of things to try to rule out various possibilities and
we're left with what seems like an improbability of a basic problem on the
Cassandra server.

Just to close, we're running nodetool repair nightly (right after we do a
nodetool flush), we have GCGraceSeconds set to 2 days, and read/writes for
our tests are CL.ALL.  Should we be running nodetool compact also?

Thanks,
  Scott

Re: time to live rows

[Bill Speirs]

I don't think this is supported (but I could be completely wrong).
However, I'd love to see this functionality as well.

How would one go about requesting such a feature?

Bill-

On Mon, Feb 7, 2011 at 4:15 PM, Kallin Nagelberg
 wrote:
> Hey,
>
> I have read about the new TTL columns in Cassandra 0.7. In my case I'd
> like to expire an entire row automatically after a certain amount of
> time. Is this possible as well?
>
> Thanks,
> -Kal
>

Re: Best way to detect/fix bitrot today?

[Peter Schuller]

> Some RAID storage might do it, potentially more efficiently!!

People keep claiming that but I have yet to confirm that a hardware
raid does actual checksumming as opposed to just healing bad blocks.
But yes, they might :)

> Food for thought, or wild imagination ?

That was my intent. Checksumming at the sstable level would allow
detection of corruption, and regular repair and read-repair would
provide self-healing.

-- 
/ Peter Schuller

Re: Deleted columns still coming back; CASSANDRA-{1748,1837} alive in 0.6.x?

[Aaron Morton]

Was there a time where nodetool repair was not run frequently ?There are some steps listed here to reset issues around tombstones coming back to life http://wiki.apache.org/cassandra/Operations#Dealing_with_the_consequences_of_nodetool_repair_not_running_within_GCGraceSecondsWhy do you run nodetool flush before the repair ?Hope that helpsAaronOn 08 Feb, 2011,at 10:19 AM, Scott McCarty  wrote:Hi,Does anyone know if anything similar to https://issues.apache.org/jira/browse/CASSANDRA-1748 or https://issues.apache.org/jira/browse/CASSANDRA-1837 exists in 0.6.x releases?  Both of those bugs look like they were introduced, found, and fixed in 0.7, and CASSANDRA-1837 comments indicate that 0.6 tests passed but I wanted to double check on this because we've been seeing deleted columns reappear in our cluster.
Or, maybe if someone has an idea of what might be behind the following oddness:  columns are deleted, then about two days later (which is what we have GCGraceSeconds set to) they pop back up again, shortly after running a "nodetool repair" on all the nodes in the cluster.
One tell-tale clue is that the timestamps on the columns that the client logs show were deleted look to be the original timestamps, as they are far back in the past and our code doesn't create timestamp values in the past.  A bunch of investigation work has us almost convinced that the deleted columns are popping up in large numbers after a repair is done.
I've posted a message to this list before regarding deleted columns coming back and one suggestion received was to assume it's a client-side bug, so we've done a ton of things to try to rule out various possibilities and we're left with what seems like an improbability of a basic problem on the Cassandra server.
Just to close, we're running nodetool repair nightly (right after we do a nodetool flush), we have GCGraceSeconds set to 2 days, and read/writes for our tests are CL.ALL.  Should we be running nodetool compact also?
Thanks,  Scott

Re: time to live rows

[Aaron Morton]

Deleting all the columns in a row via TTL has the same affect as deleting th row, the data will physically by removed during compaction. AaronOn 08 Feb, 2011,at 10:24 AM, Bill Speirs  wrote:I don't think this is supported (but I could be completely wrong).
However, I'd love to see this functionality as well.

How would one go about requesting such a feature?

Bill-

On Mon, Feb 7, 2011 at 4:15 PM, Kallin Nagelberg
 wrote:
> Hey,
>
> I have read about the new TTL columns in Cassandra 0.7. In my case I'd
> like to expire an entire row automatically after a certain amount of
> time. Is this possible as well?
>
> Thanks,
> -Kal
>

Re: time to live rows

[Kallin Nagelberg]

I tried that but I still see the row coming back on a list
 in the CLI. My concern is that there will be a pointer
to an empty row for all eternity.

-Kal

On Mon, Feb 7, 2011 at 4:38 PM, Aaron Morton  wrote:
> Deleting all the columns in a row via TTL has the same affect as deleting th
> row, the data will physically by removed during compaction.
>
> Aaron
>
>
> On 08 Feb, 2011,at 10:24 AM, Bill Speirs  wrote:
>
> I don't think this is supported (but I could be completely wrong).
> However, I'd love to see this functionality as well.
>
> How would one go about requesting such a feature?
>
> Bill-
>
> On Mon, Feb 7, 2011 at 4:15 PM, Kallin Nagelberg
>  wrote:
>> Hey,
>>
>> I have read about the new TTL columns in Cassandra 0.7. In my case I'd
>> like to expire an entire row automatically after a certain amount of
>> time. Is this possible as well?
>>
>> Thanks,
>> -Kal
>>
>

Re: Deleted columns still coming back; CASSANDRA-{1748,1837} alive in 0.6.x?

[Scott McCarty]

Yes, we failed to run nodetool repair for quite a while and I believe it
might have been our situation that prompted the addition of that info to the
wiki :-)

We've tried/are trying two of the suggested steps there, but haven't done
the process of removing/reinserting the pseudo-failed nodes (all of them?).
 I'm thinking that won't make a difference as these deletes are being all
the time as the columns pop up and we automatically detect errors and remove
the columns (for the n'th time :-) so the timestamps should be much later
than any old & existing columns that aren't supposed to be there on the
nodes.  At least that's what I've convinced myself of!

As far as running flush before repair, I got that idea from the comments in
1748.  I'm not convinced at all it's necessary but thought it might help if
there was (still?) a problem in 0.6.10 code.

--Scott

On Mon, Feb 7, 2011 at 2:31 PM, Aaron Morton wrote:

> Was there a time where nodetool repair was not run frequently ?
>
> There are some steps listed here to reset issues
> around tombstones coming back to life
>
> http://wiki.apache.org/cassandra/Operations#Dealing_with_the_consequences_of_nodetool_repair_not_running_within_GCGraceSeconds
>
>
> Why
> do you run nodetool flush before the repair ?
>
> Hope that helps
> Aaron
>
>
> On 08 Feb, 2011,at 10:19 AM, Scott McCarty  wrote:
>
> Hi,
>
> Does anyone know if anything similar to
> https://issues.apache.org/jira/browse/CASSANDRA-1748 or
> https://issues.apache.org/jira/browse/CASSANDRA-1837 exists in 0.6.x
> releases?  Both of those bugs look like they were introduced, found, and
> fixed in 0.7, and CASSANDRA-1837 comments indicate that 0.6 tests passed but
> I wanted to double check on this because we've been seeing deleted columns
> reappear in our cluster.
>
> Or, maybe if someone has an idea of what might be behind the following
> oddness:  columns are deleted, then about two days later (which is what we
> have GCGraceSeconds set to) they pop back up again, shortly after running a
> "nodetool repair" on all the nodes in the cluster.
>
> One tell-tale clue is that the timestamps on the columns that the client
> logs show were deleted look to be the original timestamps, as they are far
> back in the past and our code doesn't create timestamp values in the past.
>  A bunch of investigation work has us almost convinced that the deleted
> columns are popping up in large numbers after a repair is done.
>
> I've posted a message to this list before regarding deleted columns coming
> back and one suggestion received was to assume it's a client-side bug, so
> we've done a ton of things to try to rule out various possibilities and
> we're left with what seems like an improbability of a basic problem on the
> Cassandra server.
>
> Just to close, we're running nodetool repair nightly (right after we do a
> nodetool flush), we have GCGraceSeconds set to 2 days, and read/writes for
> our tests are CL.ALL.  Should we be running nodetool compact also?
>
> Thanks,
>   Scott
>
>

Re: time to live rows

[Kallin Nagelberg]

I also tried forcing a major compaction on the column family using JMX
but the row remains.

On Mon, Feb 7, 2011 at 4:43 PM, Kallin Nagelberg
 wrote:
> I tried that but I still see the row coming back on a list
>  in the CLI. My concern is that there will be a pointer
> to an empty row for all eternity.
>
> -Kal
>
> On Mon, Feb 7, 2011 at 4:38 PM, Aaron Morton  wrote:
>> Deleting all the columns in a row via TTL has the same affect as deleting th
>> row, the data will physically by removed during compaction.
>>
>> Aaron
>>
>>
>> On 08 Feb, 2011,at 10:24 AM, Bill Speirs  wrote:
>>
>> I don't think this is supported (but I could be completely wrong).
>> However, I'd love to see this functionality as well.
>>
>> How would one go about requesting such a feature?
>>
>> Bill-
>>
>> On Mon, Feb 7, 2011 at 4:15 PM, Kallin Nagelberg
>>  wrote:
>>> Hey,
>>>
>>> I have read about the new TTL columns in Cassandra 0.7. In my case I'd
>>> like to expire an entire row automatically after a certain amount of
>>> time. Is this possible as well?
>>>
>>> Thanks,
>>> -Kal
>>>
>>
>

unique key generation

[Kallin Nagelberg]

Hey,

I am developing a session management system using Cassandra and need
to generate unique sessionIDs (cassandra columnfamily keys). Does
anyone know of an elegant/simple way to accomplish this? I am not sure
about using time based uuids on the client as there a chance that
multiple clients could generate the same ID. I've heard suggestions of
using zookeeper as a source for the IDs, but was just hoping that
there might be something simpler for my purposes.

Thanks,
-Kal

Re: [0.7.1] more exceptions: Illegal mode

[Aaron Morton]

I noticed this as well in a machine that was left running with the current 0.7 branch code. Created https://issues.apache.org/jira/browse/CASSANDRA-2131aaronOn 08 Feb, 2011,at 04:34 AM, Jake Luciani  wrote:This sounds like a possible bug since the BRAF was re-written in 0.7.1. Could you open a ticket?On Mon, Feb 7, 2011 at 10:32 AM, Patrik Modesto  wrote:
On Mon, Feb 7, 2011 at 15:42, Thibaut Britz
 wrote:
> I think this is related to a faulty disk.

I'm not sure thats the problem. Cassandra 0.7.0 didn't report any
problem. It started with Cassandra 0.7.1.

Patrik

Re: unique key generation

[Victor Kabdebon]

Hello Kallin.
If you use timeUUID the chance to generate two time the same uuid is the
following :
considering that both client generate the uuid at the *same millisecond*,
the chance of generating the same uuid is :

1/1.84467441 × 1019Which is equal to the probability for winning a national
lottery for 1e11 days in a row ( for 270 million years).
Well if you do have a collision you should play the lottery :).

Best regards,
Victor Kabdebon
http://www.voxnucleus.fr

2011/2/7 Kallin Nagelberg 

> Hey,
>
> I am developing a session management system using Cassandra and need
> to generate unique sessionIDs (cassandra columnfamily keys). Does
> anyone know of an elegant/simple way to accomplish this? I am not sure
> about using time based uuids on the client as there a chance that
> multiple clients could generate the same ID. I've heard suggestions of
> using zookeeper as a source for the IDs, but was just hoping that
> there might be something simpler for my purposes.
>
> Thanks,
> -Kal
>

Re: unique key generation

[Kallin Nagelberg]

Maybe I can just use java5's UUID.. Need to research how this is
effective across multiple clients..

On Mon, Feb 7, 2011 at 4:57 PM, Kallin Nagelberg
 wrote:
> Hey,
>
> I am developing a session management system using Cassandra and need
> to generate unique sessionIDs (cassandra columnfamily keys). Does
> anyone know of an elegant/simple way to accomplish this? I am not sure
> about using time based uuids on the client as there a chance that
> multiple clients could generate the same ID. I've heard suggestions of
> using zookeeper as a source for the IDs, but was just hoping that
> there might be something simpler for my purposes.
>
> Thanks,
> -Kal
>

Re: order of index expressions

[Jonathan Ellis]

On Sun, Feb 6, 2011 at 11:03 AM, Shaun Cutts  wrote:
> What I think you should be doing is the following: open iterators on the 
> matching keys for each of the indexes; the inside loop would pick an iterator 
> at random, and pull a match from it. This would assure that the expected 
> number of entries examined is a small multiple (# of other indexes) of the 
> index with the most "precision".

Isn't this a bad approximation of performing an intersection by
iterating through each index results (which, since they return in
sorted order, can be done without horrible memory cost)?

I'm not opposed to doing that.  But I have no idea how to guess when
that is better than the current inner-loop method.

> I know you have a new type of index in the works... but it doesn't look like 
> "trunk" has any modifications for "scan", and presumably the strategy I just 
> mentioned is pretty general (not depending on histograms, etc). Does it sound 
> like a good idea?

I think you're thinking of
https://issues.apache.org/jira/browse/CASSANDRA-1472, and my
understanding is that the bitmap approach basically does what you want
only faster.

-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Re: seed node failure crash the whole cluster

[Jonathan Ellis]

On Mon, Feb 7, 2011 at 1:51 AM, TSANG Yiu Wing  wrote:
> cassandra version: 0.7
>
> client library: scale7-pelops / 1.0-RC1-0.7.0-SNAPSHOT
>
> cluster: 3 machines (A, B, C)
>
> details:
> it works perfectly when all 3 machines are up and running
>
> but if the seed machine is down, the problems happen:
>
> 1) new client connection cannot be established

sounds like pelops relies on the seed node to introduce it to the
cluster.  you should configure it either with a hardcoded list of
nodes or use something like RRDNS instead.  I don't use pelops so I
can't help other than that.  (I believe there is a mailing list for
Pelops though.)

> 2) if a client keeps connecting to and operating at (issue get and
> update) the cluster, when the seed is down, the working client will
> throw exception upon the next operation

I know Hector supports transparent failover to another Cassandra node.
 Perhaps Pelops does not.

> 3) using cassandra-cli to connect the remaining nodes in the cluster,
> "Internal error processing get_range_slices" will happen when querying
> column family
>> list ;

Cassandra always logs the cause of internal errors in system.log, so
you should look there.

-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Re: [0.7.1] Error in ThreadPoolExecutor

[Jonathan Ellis]

Can you open a ticket for this?  And are you using order-preserving partitioner?

On Mon, Feb 7, 2011 at 7:16 AM, Patrik Modesto  wrote:
> Hi,
>
> on my two-node test setup I get repeatedly following error:
>
> The 10.0.18.129 server log:
>
>  INFO 14:10:37,707 Node /10.0.18.99 has restarted, now UP again
>  INFO 14:10:37,708 Checking remote schema before delivering hints
>  INFO 14:10:37,708 Sleeping 45506ms to stagger hint delivery
>  INFO 14:10:37,709 Node /10.0.18.99 state jump to normal
>  INFO 14:11:23,215 Started hinted handoff for endpoint /10.0.18.99
> ERROR 14:11:23,884 Error in ThreadPoolExecutor
> java.lang.RuntimeException: java.lang.IllegalArgumentException
>        at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>        at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.IllegalArgumentException
>        at java.nio.Buffer.position(Buffer.java:218)
>        at 
> org.apache.cassandra.utils.ByteBufferUtil.string(ByteBufferUtil.java:117)
>        at 
> org.apache.cassandra.utils.ByteBufferUtil.string(ByteBufferUtil.java:111)
>        at 
> org.apache.cassandra.db.HintedHandOffManager.getTableAndCFNames(HintedHandOffManager.java:237)
>        at 
> org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:306)
>        at 
> org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:88)
>        at 
> org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:385)
>        at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>        ... 3 more
> ERROR 14:11:23,885 Fatal exception in thread Thread[HintedHandoff:1,1,main]
> java.lang.RuntimeException: java.lang.IllegalArgumentException
>        at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>        at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.IllegalArgumentException
>        at java.nio.Buffer.position(Buffer.java:218)
>        at 
> org.apache.cassandra.utils.ByteBufferUtil.string(ByteBufferUtil.java:117)
>        at 
> org.apache.cassandra.utils.ByteBufferUtil.string(ByteBufferUtil.java:111)
>        at 
> org.apache.cassandra.db.HintedHandOffManager.getTableAndCFNames(HintedHandOffManager.java:237)
>        at 
> org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:306)
>        at 
> org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:88)
>        at 
> org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:385)
>        at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>        ... 3 more
>
>
> The 10.0.18.99 server log:
>
>  INFO 14:10:37,691 Binding thrift service to /0.0.0.0:9160
>  INFO 14:10:37,693 Using TFastFramedTransport with a max frame size of
> 15728640 bytes.
>  INFO 14:10:37,695 Listening for thrift clients...
>  INFO 14:10:38,337 GC for ParNew: 954 ms, 658827608 reclaimed leaving
> 966732432 used; max is 4265607168
>  INFO 14:11:27,142 Started hinted handoff for endpoint /10.0.18.129
> ERROR 14:11:27,370 Error in ThreadPoolExecutor
> java.lang.RuntimeException: java.lang.IllegalArgumentException
>        at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>        at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.IllegalArgumentException
>        at java.nio.Buffer.position(Buffer.java:218)
>        at 
> org.apache.cassandra.utils.ByteBufferUtil.string(ByteBufferUtil.java:117)
>        at 
> org.apache.cassandra.utils.ByteBufferUtil.string(ByteBufferUtil.java:111)
>        at 
> org.apache.cassandra.db.HintedHandOffManager.getTableAndCFNames(HintedHandOffManager.java:237)
>        at 
> org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:306)
>        at 
> org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:88)
>        at 
> org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:385)
>        at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>        ... 3 more
> ERROR 14:11:27,371 Fatal exception in thread Thread[HintedHandoff:1,1,main]
> java.lang.RuntimeException: java.lang.IllegalArgumentException
>        at 
> org.apache.cassandra.utils.WrappedRunnable

Re: OOM during batch_mutate

[Jonathan Ellis]

Sounds like the keyspace was created on the 32GB machine, so it
guessed memtable sizes that are too large when run on the 16GB one.
Use "update column family" from the cli to cut the throughput and
operations thresholds in half, or to 1/4 to be cautious.

On Mon, Feb 7, 2011 at 9:00 AM, Patrik Modesto  wrote:
> On Mon, Feb 7, 2011 at 15:44, sridhar basam  wrote:
>> Looks like you don't have a big enough working set from your GC logs, there
>> doesn't seem to be a lot being reclaimed in the GC process. The process is
>> reclaiming a few hundred MB and is running every few seconds. How big are
>> your caches? The probable reason that it works the first couple times when
>> you create it due to nothing being in cache as it gets built up.
>
> Hi Sridhar,
>
> thanks for your answer. I didn't set the cache to any specific value,
> I use the defaults:
>
> [default@Foo] show keyspaces;
> Keyspace: Foo:
>  Replication Strategy: org.apache.cassandra.locator.SimpleStrategy
>    Replication Factor: 1
>  Column Families:
>    ColumnFamily: Url
>      Columns sorted by: org.apache.cassandra.db.marshal.UTF8Type
>      Row cache size / save period: 0.0/0
>      Key cache size / save period: 20.0/3600
>      Memtable thresholds: 4.6640625/995/60
>      GC grace seconds: 864000
>      Compaction min/max thresholds: 4/32
>      Read repair chance: 1.0
>      Built indexes: []
>
> My row-key is 16bytes long string, so I don't expect the cache would
> be a problem.
>
> Patrik
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Re: Java bombs during compaction, please help

[Jonathan Ellis]

I've patched ColumnSortedMap on the 0.7 branch to not swallow the
IOException it's getting.

On Mon, Feb 7, 2011 at 3:02 PM, buddhasystem  wrote:
>
> Hello,
> one node in my 3-machine cluster cannot perform compaction. I tried multiple
> times, it ran out of heap space once and I increased it. Now I'm getting the
> dump below (after it does run for a few minutes). I hope somebody can shed a
> little light on what' going on, because I'm at a loss and this is a real
> show stopper.
>
>
> [me@mymachine]~/cassandra-test% Error occured while compacting keyspace
> Tracer
> java.util.concurrent.ExecutionException: java.lang.NullPointerException
>        at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source)
>        at java.util.concurrent.FutureTask.get(Unknown Source)
>        at
> org.apache.cassandra.db.CompactionManager.performMajor(CompactionManager.java:186)
>        at
> org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1766)
>        at
> org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1236)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
>        at java.lang.reflect.Method.invoke(Unknown Source)
>        at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
>        at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
>        at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source)
>        at com.sun.jmx.mbeanserver.PerInterface.invoke(Unknown Source)
>        at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source)
>        at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown Source)
>        at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)
>        at javax.management.remote.rmi.RMIConnectionImpl.doOperation(Unknown
> Source)
>        at javax.management.remote.rmi.RMIConnectionImpl.access$200(Unknown
> Source)
>        at
> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(Unknown
> Source)
>        at
> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(Unknown
> Source)
>        at javax.management.remote.rmi.RMIConnectionImpl.invoke(Unknown
> Source)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
>        at java.lang.reflect.Method.invoke(Unknown Source)
>        at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
>        at sun.rmi.transport.Transport$1.run(Unknown Source)
>        at java.security.AccessController.doPrivileged(Native Method)
>        at sun.rmi.transport.Transport.serviceCall(Unknown Source)
>        at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
>        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown
> Source)
>        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown
> Source)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
> Source)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
>        at java.lang.Thread.run(Unknown Source)
> Caused by: java.lang.NullPointerException
>        at
> org.apache.cassandra.io.util.ColumnIterator$1.getKey(ColumnSortedMap.java:276)
>        at
> org.apache.cassandra.io.util.ColumnIterator$1.getKey(ColumnSortedMap.java:263)
>        at
> java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(Unknown Source)
>        at java.util.concurrent.ConcurrentSkipListMap.(Unknown Source)
>        at
> org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:384)
>        at
> org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:332)
>        at
> org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129)
>        at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:137)
>        at
> org.apache.cassandra.io.PrecompactedRow.(PrecompactedRow.java:78)
>        at
> org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:139)
>        at
> org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:108)
>        at
> org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:43)
>        at
> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
>        at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
>        at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
>        at
> org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterato

Re: seed node failure crash the whole cluster

[Dan Washusen]

Hi,
I've added some comments and questions inline.

Cheers,
Dan

On 8 February 2011 10:00, Jonathan Ellis  wrote:

> On Mon, Feb 7, 2011 at 1:51 AM, TSANG Yiu Wing  wrote:
> > cassandra version: 0.7
> >
> > client library: scale7-pelops / 1.0-RC1-0.7.0-SNAPSHOT
> >
> > cluster: 3 machines (A, B, C)
> >
> > details:
> > it works perfectly when all 3 machines are up and running
> >
> > but if the seed machine is down, the problems happen:
> >
> > 1) new client connection cannot be established
>
> sounds like pelops relies on the seed node to introduce it to the
> cluster.  you should configure it either with a hardcoded list of
> nodes or use something like RRDNS instead.  I don't use pelops so I
> can't help other than that.  (I believe there is a mailing list for
> Pelops though.)
>

When dynamic node discovery is turned on (off by default) it doesn't
(shouldn't) rely on the initial seed node once past initialization.  So
either make sure you have dynamic node discovery turned on or seed Pelops
with all nodes in your cluster...

It would be helpful if you provided more information about the errors you're
seeing preferably with debug level logging turned on.


>
> > 2) if a client keeps connecting to and operating at (issue get and
> > update) the cluster, when the seed is down, the working client will
> > throw exception upon the next operation
>
> I know Hector supports transparent failover to another Cassandra node.
>  Perhaps Pelops does not.
>

Pelops will validate connections at a configurable period (60 seconds by
default) and remove them from the pool.  Pelops will also retry the
operation three times (configurable) against a different node in the pool
each time.

If you want Pelops to take more agressive actions when it detects downed
nodes then check out
org.scale7.cassandra.pelops.pool.CommonsBackedPool.INodeSuspensionStrategy.


>
> > 3) using cassandra-cli to connect the remaining nodes in the cluster,
> > "Internal error processing get_range_slices" will happen when querying
> > column family
> >> list ;
>
> Cassandra always logs the cause of internal errors in system.log, so
> you should look there.
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>

Re: Java bombs during compaction, please help

[buddhasystem]

Thanks Jonathan -- does it mean that the machine is experiencing IO problems?

-- 
View this message in context: 
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Java-bombs-during-compaction-please-help-tp6001773p6002320.html
Sent from the cassandra-u...@incubator.apache.org mailing list archive at 
Nabble.com.

Re: unique key generation

[Patricio Echagüe]

If you are a Hector user, TimeUUIDUtils can be used to create Time UUIDs.

https://github.com/rantav/hector/blob/master/core/src/main/java/me/prettyprint/cassandra/utils/TimeUUIDUtils.java


On Mon, Feb 7, 2011 at 2:11 PM, Kallin Nagelberg  wrote:

> Maybe I can just use java5's UUID.. Need to research how this is
> effective across multiple clients..
>
> On Mon, Feb 7, 2011 at 4:57 PM, Kallin Nagelberg
>  wrote:
> > Hey,
> >
> > I am developing a session management system using Cassandra and need
> > to generate unique sessionIDs (cassandra columnfamily keys). Does
> > anyone know of an elegant/simple way to accomplish this? I am not sure
> > about using time based uuids on the client as there a chance that
> > multiple clients could generate the same ID. I've heard suggestions of
> > using zookeeper as a source for the IDs, but was just hoping that
> > there might be something simpler for my purposes.
> >
> > Thanks,
> > -Kal
> >
>

Re: order of index expressions

[Shaun Cutts]

Jonathan,

Thanks for your thoughts

> On Sun, Feb 6, 2011 at 11:03 AM, Shaun Cutts  wrote:
>> What I think you should be doing is the following: open iterators on the 
>> matching keys for each of the indexes; the inside loop would pick an 
>> iterator at random, and pull a match from it. This would assure that the 
>> expected number of entries examined is a small multiple (# of other indexes) 
>> of the index with the most "precision".
> 
> Isn't this a bad approximation of performing an intersection by
> iterating through each index results (which, since they return in
> sorted order, can be done without horrible memory cost)?

I don't think so, unless I'm not understanding you. The point is... suppose you 
have 1M matches on the key for one index, and 1 in the other (for a row among 
the 1M matching the first), and you have no information about which is which. 
If you arbitrarily pick one to go first at random, you loop 500K times on 
average.

If you do what I'm suggesting you loop 2 times on average -- a big difference 
:). This is "breadth first search" but the breadth is just the number of 
indexes... if there are really thousands of indexes you could bound to the X 
most likely, but in that case getting the order right for optimal depth-first 
is quite unlikely, whereas since this is an "AND", for even modest X you lower 
your chances of a "wrong" choice exponentially.

Also, is the amount of memory for a "cursor" in an index really that large 
compared with the memory that you use for query specification in any case? Your 
current method does assume it has random access to the query.

I do think its a flaw that something which can be done in constant time 
actually takes time which could be proportional to the number of rows, just by 
picking the wrong index.

> I'm not opposed to doing that.  But I have no idea how to guess when
> that is better than the current inner-loop method.

What I'm proposing is always at most a constant times number of iterations 
worse (ok -- # of times proportional to the number of indexes used), vs 
something that could be arbitrarily much worse. But you could adapt the current 
heuristic to tune the probabilities, so that the worse case will be 
asymptotically the same as current but the average case will be much better.

> 
>> I know you have a new type of index in the works... but it doesn't look like 
>> "trunk" has any modifications for "scan", and presumably the strategy I just 
>> mentioned is pretty general (not depending on histograms, etc). Does it 
>> sound like a good idea?
> 
> I think you're thinking of
> https://issues.apache.org/jira/browse/CASSANDRA-1472, and my
> understanding is that the bitmap approach basically does what you want
> only faster.
> 
I'll have to study it. As far as I know what I'm proposing is orthogonal to 
histogram methods that don't actually keep stats on the joins (which is a lot 
of stats to keep), and even in this case it can be used to optimize "lower 
resolution" statistics to the partial experience from sampling the current 
query. 

It does introduce extra overhead for small queries, of course, so if you had 
histograms that guaranteed that the query was going to be very small for a 
particular index then it would be best to put that first without fiddling. 
However, typically you have exact info for only the common keys, not the rare 
ones.

-- Shaun

> -- 
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com

Re: Deleted columns still coming back; CASSANDRA-{1748,1837} alive in 0.6.x?

[Peter Schuller]

> Yes, we failed to run nodetool repair for quite a while and I believe it
> might have been our situation that prompted the addition of that info to the
> wiki :-)

So is it correct then that nowadays, you hope to not be violating the
repair frequency requirement but you're still seeing data pop up since
after the point at which you "fixed" this?

One thing is: I presume you have the daily job running the repair. Is
it possible some of those have somehow failed? If the repair is daily
and grace is 2 days, even a single sporadic repair may cause a
violation of the repair interval requirement.

-- 
/ Peter Schuller

unsubscribe

[mike dooley]

unsubscribe

Re: Do supercolumns have a purpose?

[Shaun Cutts]

I'm a newbie here, but, with apologies for my presumptuousness, I think you 
should deprecate SuperColumns. They are already distracting you, and as the 
years go by the cost of supporting them as you add more and more functionality 
is only likely to get worse. It would be better to concentrate on making the 
"core" column families better (and I'm sure we can all think of lots of things 
we'd like).

Just dropping SuperColumns would be bad for your reputation -- and for users 
like David who are currently using them. But if you mark them clearly as 
deprecated and explain why and what to do instead (perhaps putting a bit of 
effort into migration tools... or even a "virtual" layer supporting arbitrary 
hierarchical data), then you can drop them in a few years (when you get to 1.0, 
say), without people feeling betrayed.

-- Shaun

On Feb 6, 2011, at 3:48 AM, David Boxenhorn wrote:

> "My main point was to say that it's think it is better to create tickets for 
> what you want, rather than for something else completely different that 
> would, as a by-product, give you what you want."
> 
> Then let me say what I want: I want supercolumn families to have any feature 
> that regular column families have. 
> 
> My data model is full of supercolumns. I used them, even though I knew it 
> didn't *have to*, "because they were there", which implied to me that I was 
> supposed to use them for some good reason. Now I suspect that they will 
> gradually become less and less functional, as features are added to regular 
> column families and not supported for supercolumn families. 
> 
> 
> On Fri, Feb 4, 2011 at 10:58 AM, Sylvain Lebresne  
> wrote:
> On Fri, Feb 4, 2011 at 12:35 AM, Mike Malone  wrote:
> On Thu, Feb 3, 2011 at 6:44 AM, Sylvain Lebresne  wrote:
> On Thu, Feb 3, 2011 at 3:00 PM, David Boxenhorn  wrote:
> The advantage would be to enable secondary indexes on supercolumn families.
> 
> Then I suggest opening a ticket for adding secondary indexes to supercolumn 
> families and voting on it. This will be 1 or 2 order of magnitude less work 
> than getting rid of super column internally, and probably a much better 
> solution anyway.
> 
> I realize that this is largely subjective, and on such matters code speaks 
> louder than words, but I don't think I agree with you on the issue of which 
> alternative is less work, or even which is a better solution.
> 
> You are right, I put probably too much emphase in that sentence. My main 
> point was to say that it's think it is better to create tickets for what you 
> want, rather than for something else completely different that would, as a 
> by-product, give you what you want.
> Then I suspect that *if* the only goal is to get secondary indexes on super 
> columns, then there is a good chance this would be less work than getting rid 
> of super columns. But to be fair, secondary indexes on super columns may not 
> make too much sense without #598, which itself would require quite some work, 
> so clearly I spoke a bit quickly.
>  
> If the goal is to have a hierarchical model, limiting the depth to two seems 
> arbitrary. Why not go all the way and allow an arbitrarily deep hierarchy?
> 
> If a more sophisticated hierarchical model is deemed unnecessary, or 
> impractical, allowing a depth of two seems inconsistent and unnecessary. It's 
> pretty trivial to overlay a hierarchical model on top of the 
> map-of-sorted-maps model that Cassandra implements. Ed Anuff has implemented 
> a custom comparator that does the job [1]. Google's Megastore has a similar 
> architecture and goes even further [2].
> 
> It seems to me that super columns are a historical artifact from Cassandra's 
> early life as Facebook's inbox storage system. They needed posting lists of 
> messages, sharded by user. So that's what they built. In my dealings with the 
> Cassandra code, super columns end up making a mess all over the place when 
> algorithms need to be special cased and branch based on the 
> column/supercolumn distinction.
> 
> I won't even mention what it does to the thrift interface.
> 
> Actually, I agree with you, more than you know. If I were to start coding 
> Cassandra now, I wouldn't include super columns (and I would probably not go 
> for a depth unlimited hierarchical model either). But it's there and I'm not 
> sure getting rid of them fully (meaning, including in thrift) is an option 
> (it would be a big compatibility breakage). And (even though I certainly 
> though about this more than once :)) I'm slightly less enthusiastic about 
> keeping them in thrift but encoding them in regular column family internally: 
> it would still be a lot of work but we would still probably end up with nasty 
> tricks to stick to the thrift api. 
>  
> --
> Sylvain
> 
> 
> Mike
> 
> [1] http://www.anuff.com/2010/07/secondary-indexes-in-cassandra.html
> [2] http://www.cidrdb.org/cidr2011/Papers/CIDR11_Paper32.pdf
> 
> 

Re: Ruby thrift is trying to write Time as string

[Joshua Partogi]

Thanks Ryan.

That makes more sense now. So I should instead find a way to (de)serialize
Ruby objects to string vice versa when inserting to Column.

Kind regards,
Joshua

On Tue, Feb 8, 2011 at 4:43 AM, Ryan King  wrote:

> On Sat, Feb 5, 2011 at 10:12 PM, Joshua Partogi 
> wrote:
> > Hi,
> >
> > I don't know whether my assumption is right or not. When I tried to
> insert a
> > Time value into a column I am getting this exception:
> >
> >
> vendor/ruby/1.8/gems/thrift-0.5.0/lib/thrift/protocol/binary_protocol.rb:106:in
> > `write_string'
> > vendor/ruby/1.8/gems/thrift-0.5.0/lib/thrift/client.rb:35:in `write'
> > vendor/ruby/1.8/gems/thrift-0.5.0/lib/thrift/client.rb:35:in
> `send_message'
> >
> vendor/ruby/1.8/gems/cassandra-0.9.0/lib/./vendor/0.7/gen-rb/cassandra.rb:213:in
> > `send_batch_mutate'
> >
> vendor/ruby/1.8/gems/cassandra-0.9.0/lib/./vendor/0.7/gen-rb/cassandra.rb:208:in
> > `batch_mutate'
> >
> vendor/ruby/1.8/gems/thrift_client-0.6.0/lib/thrift_client/abstract_thrift_client.rb:115:in
> > `send'
> >
> vendor/ruby/1.8/gems/thrift_client-0.6.0/lib/thrift_client/abstract_thrift_client.rb:115:in
> > `handled_proxy'
> >
> vendor/ruby/1.8/gems/thrift_client-0.6.0/lib/thrift_client/abstract_thrift_client.rb:57:in
> > `batch_mutate'
> > vendor/ruby/1.8/gems/cassandra-0.9.0/lib/cassandra/0.7/protocol.rb:8:in
> > `_mutate'
> > vendor/ruby/1.8/gems/cassandra-0.9.0/lib/cassandra/cassandra.rb:130:in
> > `insert'
> >
> > But I am not getting any error if I insert a Time value into a
> sub-column.
> >
> > Is this an error or does it suppose to work that way?
> >
> > Thanks heaps for the insight.
>
> IIRC when we wrote that code the idea was to require strings for
> everything and have users do their own serialization. In that context,
> the supercolmn behavior is the wrong one here.
>
> -ryan
>
> --
> @rk
>



-- 
http://twitter.com/jpartogi

Re: seed node failure crash the whole cluster

[TSANG Yiu Wing]

i will continue the issue here:

http://groups.google.com/group/scale7/browse_thread/thread/dd74f1d6265ae2e7

thanks


On Tue, Feb 8, 2011 at 7:44 AM, Dan Washusen  wrote:
> Hi,
> I've added some comments and questions inline.
>
> Cheers,
> Dan
> On 8 February 2011 10:00, Jonathan Ellis  wrote:
>>
>> On Mon, Feb 7, 2011 at 1:51 AM, TSANG Yiu Wing  wrote:
>> > cassandra version: 0.7
>> >
>> > client library: scale7-pelops / 1.0-RC1-0.7.0-SNAPSHOT
>> >
>> > cluster: 3 machines (A, B, C)
>> >
>> > details:
>> > it works perfectly when all 3 machines are up and running
>> >
>> > but if the seed machine is down, the problems happen:
>> >
>> > 1) new client connection cannot be established
>>
>> sounds like pelops relies on the seed node to introduce it to the
>> cluster.  you should configure it either with a hardcoded list of
>> nodes or use something like RRDNS instead.  I don't use pelops so I
>> can't help other than that.  (I believe there is a mailing list for
>> Pelops though.)
>
> When dynamic node discovery is turned on (off by default) it doesn't
> (shouldn't) rely on the initial seed node once past initialization.  So
> either make sure you have dynamic node discovery turned on or seed Pelops
> with all nodes in your cluster...
> It would be helpful if you provided more information about the errors you're
> seeing preferably with debug level logging turned on.
>
>>
>> > 2) if a client keeps connecting to and operating at (issue get and
>> > update) the cluster, when the seed is down, the working client will
>> > throw exception upon the next operation
>>
>> I know Hector supports transparent failover to another Cassandra node.
>>  Perhaps Pelops does not.
>
> Pelops will validate connections at a configurable period (60 seconds by
> default) and remove them from the pool.  Pelops will also retry the
> operation three times (configurable) against a different node in the pool
> each time.
> If you want Pelops to take more agressive actions when it detects downed
> nodes then check out
> org.scale7.cassandra.pelops.pool.CommonsBackedPool.INodeSuspensionStrategy.
>
>>
>> > 3) using cassandra-cli to connect the remaining nodes in the cluster,
>> > "Internal error processing get_range_slices" will happen when querying
>> > column family
>> >> list ;
>>
>> Cassandra always logs the cause of internal errors in system.log, so
>> you should look there.
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of DataStax, the source for professional Cassandra support
>> http://www.datastax.com
>
>

Re: unique key generation

[Kallin Nagelberg]

Pretty sure it also uses mac address, so chances are very slim. I'll check
out time uuid too, thanks.

On 7 Feb 2011 17:11, "Victor Kabdebon"  wrote:

Hello Kallin.
If you use timeUUID the chance to generate two time the same uuid is the
following :
considering that both client generate the uuid at the *same millisecond*,
the chance of generating the same uuid is :

1/1.84467441 × 1019Which is equal to the probability for winning a national
lottery for 1e11 days in a row ( for 270 million years).
Well if you do have a collision you should play the lottery :).

Best regards,
Victor Kabdebon
http://www.voxnucleus.fr

2011/2/7 Kallin Nagelberg 


>
> Hey,
>
> I am developing a session management system using Cassandra and need
> to generate uni...

Cassandra memory consumption

[Victor Kabdebon]

Dear all,

Sorry to come back again to this point but I am really worried about
Cassandra memory consumption. I have a single machine that runs one
Cassandra server. There is almost no data on it but I see a crazy memory
consumption and it doesn't care at all about the instructions...
Note that I am not using mmap, but "Standard", I use also JNA (inside lib
folder), i am running on debian 5 64 bits, so a pretty normal configuration.
I also use Cassandra 0.6.8.


Here are the informations I gathered on Cassandra :

105  16765  0.1 34.1 1089424* 687476* ?  Sl   Feb02  14:58
/usr/bin/java -ea* -Xms128M* *-Xmx256M* -XX:+UseParNewGC
-XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8
-XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError
-Dcom.sun.management.jmxremote.port=8081
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dstorage-config=bin/../conf -Dcassandra-foreground=yes -cp
bin/../conf:bin/../build/classes:bin/../lib/antlr-3.1.3.jar:bin/../lib/apache-cassandra-0.6.6.jar:bin/../lib/avro-1.2.0-dev.jar:bin/../lib/cassandra-javautils.jar:bin/../lib/clhm-production.jar:bin/../lib/commons-cli-1.1.jar:bin/../lib/commons-codec-1.2.jar:bin/../lib/commons-collections-3.2.1.jar:bin/../lib/commons-io-1.4.jar:bin/../lib/commons-lang-2.4.jar:bin/../lib/commons-pool-1.5.4.jar:bin/../lib/google-collections-1.0.jar:bin/../lib/hadoop-core-0.20.1.jar:bin/../lib/hector-0.6.0-14.jar:bin/../lib/high-scale-lib.jar:bin/../lib/ivy-2.1.0.jar:bin/../lib/jackson-core-asl-1.4.0.jar:bin/../lib/jackson-mapper-asl-1.4.0.jar:bin/../lib/jline-0.9.94.jar:bin/../lib/jna.jar:bin/../lib/json-simple-1.1.jar:bin/../lib/libthrift-r917130.jar:bin/../lib/log4j-1.2.14.jar:bin/../lib/perf4j-0.9.12.jar:bin/../lib/slf4j-api-1.5.8.jar:bin/../lib/slf4j-log4j12-1.5.8.jar:bin/../lib/uuid-3.1.jar
org.apache.cassandra.thrift.CassandraDaemon

result of nodetool info :

116024732779488843382476400091948985708
*Load : 1,94 MB*
Generation No: 1296673772
Uptime (seconds) : 467550
*Heap Memory (MB) : 120,26 / 253,94*


I have about 21 column families, none of them have a lot of information ( as
you see I have 2 Mb of text which is really small). Even if I set Xmx at 256
there is 687M of memory used. Where does this memory come from ? Bad garbage
collection ? Something that I ignore ?
Thank you for your help I really need to get rid of that problem.

Best regards,
Victor Kabdebon

Best Approaches for Developer Integration

[Paul Querna]

Hi,

Lets suppose you are using Cassandra happily in production, but you
have an army of coders, with varying levels of knowledge about
Cassandra.

Currently we have hid most of our developers from the Cassandra
dependency by using a Fake interface that returns fake data from it,
but this is turning out to be less than ideal.

For example, for CouchDB has CouchDBX 
which at least on OSX present a very easy to use installer, data
browser, and GUI.  You just run CouchDBX.app, and then your
application can build out the rest of your data as needed for
development.

So, I guess this is coming down to:
  1) Has anyone built any easy to install packages of Cassandra?

  2) Can anyone explain their experience with getting non-Cassandra
developers up and running in a development environment? What worked?
What was hard?

Thanks,

Paul

Re: Best Approaches for Developer Integration

[Paul Brown]

On Feb 7, 2011, at 10:28 PM, Paul Querna wrote:
> So, I guess this is coming down to:
>  1) Has anyone built any easy to install packages of Cassandra?

I didn't find it necessary.  I implemented a simple embedding wrapper for 
Cassandra so that it could be started as part of a web application lifecycle 
(Spring-managed).  Developers just start up a personal copy of the service as 
part of "mvn -Pembedded-cassandra jetty:run" being none the wiser about the 
Cassandra underneath unless they care to be.

>  2) Can anyone explain their experience with getting non-Cassandra
> developers up and running in a development environment? What worked?
> What was hard?

I've had technically savvy non-developer resources perfectly happy to work with 
the system via Ruby, PHP, or even the Cassandra CLI.  "Just do mvn 
-Pembedded-cassandra jetty:run" was too much in that case, but "Here are some 
useful libraries and here's where the prod/staging clusters are" was fine.

-- Paul B

Re: [0.7.1] Error in ThreadPoolExecutor

[Patrik Modesto]

Hi,

here is the ticket: https://issues.apache.org/jira/browse/CASSANDRA-2134

I'm using the default partitioner, that should be the RandomPartitioner.

HTH,
Patrik


On Tue, Feb 8, 2011 at 00:03, Jonathan Ellis  wrote:
> Can you open a ticket for this?  And are you using order-preserving 
> partitioner?

Re: time to live rows

[Stu Hood]

The expired columns were converted into tombstones, which will live for the
GC timeout. The "empty" row will be cleaned up when those tombstones are
removed.

Returning the empty row is unfortunate... we'd love to find a more
appropriate solution that might not involve endless scanning.

See
http://wiki.apache.org/cassandra/FAQ#i_deleted_what_gives
http://wiki.apache.org/cassandra/FAQ#range_ghosts


On Mon, Feb 7, 2011 at 1:49 PM, Kallin Nagelberg  wrote:

> I also tried forcing a major compaction on the column family using JMX
> but the row remains.
>
> On Mon, Feb 7, 2011 at 4:43 PM, Kallin Nagelberg
>  wrote:
> > I tried that but I still see the row coming back on a list
> >  in the CLI. My concern is that there will be a pointer
> > to an empty row for all eternity.
> >
> > -Kal
> >
> > On Mon, Feb 7, 2011 at 4:38 PM, Aaron Morton 
> wrote:
> >> Deleting all the columns in a row via TTL has the same affect as
> deleting th
> >> row, the data will physically by removed during compaction.
> >>
> >> Aaron
> >>
> >>
> >> On 08 Feb, 2011,at 10:24 AM, Bill Speirs  wrote:
> >>
> >> I don't think this is supported (but I could be completely wrong).
> >> However, I'd love to see this functionality as well.
> >>
> >> How would one go about requesting such a feature?
> >>
> >> Bill-
> >>
> >> On Mon, Feb 7, 2011 at 4:15 PM, Kallin Nagelberg
> >>  wrote:
> >>> Hey,
> >>>
> >>> I have read about the new TTL columns in Cassandra 0.7. In my case I'd
> >>> like to expire an entire row automatically after a certain amount of
> >>> time. Is this possible as well?
> >>>
> >>> Thanks,
> >>> -Kal
> >>>
> >>
> >
>