Re: Unsubscribe

[oualid ait wafli]

Whyy ?


2013/3/18 Jeffrey Fass 

> Unsubscribe
>

Re: Pig / Map Reduce on Cassandra

[cscetbon.ext]

It succeeds but returns nothing as my columnfamily has only data for columns 
appearing in the CREATE TABLE order. It you want to keep it you should provide 
the CREATE order with sample data

--
Cyril SCETBON

On Mar 14, 2013, at 2:16 PM, aaron morton 
mailto:aa...@thelastpickle.com>> wrote:

Did the example work as it was presented in the README.txt ?

Cheers

-
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 13/03/2013, at 11:31 AM, 
cscetbon@orange.com wrote:

Finally I've found the answer in CassandraStorage.java !

columns is not an alias but a bag that you fill with columns (name+value) that 
don't have metadata.

That's why your sample doesn't return anything in my test as I've only filled 
existing columns (found in the CQL CREATE command)
I think you should update the file cassandra/examples/pig/README.txt to explain 
where this columns comes from or by changing the description of the test cause 
it doesn't really determine the top 50 column names but only the top 50 column 
names put in the bag "columns" (without metadata)

Regards
--
Cyril SCETBON

On Mar 13, 2013, at 6:44 PM, 
cscetbon@orange.com wrote:

I'm trying to execute your sample pig script and I don't understand where the 
alias "columns" comes from :


grunt> rows = LOAD 'cassandra://MyKeyspace/MyColumnFamily' USING 
CassandraStorage();
grunt> cols = FOREACH rows GENERATE flatten(columns);

I suppose it's defined by the call to getSchema function in 
src/java/org/apache/cassandra/hadoop/pig/CassandraStorage.java however even if 
the order is well executed by Pig, cols can't be dumped.

if I dump rows I get :

(iza,(),(),(),(birth_year,77),(gender,F),{})
(cyril,(),(),(),(birth_year,79),(gender,M),{})

but when I dump cols I get nothing :(

any idea ?

--
Cyril SCETBON

On Mar 13, 2013, at 10:26 AM, Cyril Scetbon 
mailto:cscetbon@orange.com>> wrote:

Ok forget it. It was a mix of mistakes like environment variables not set, 
package name not added in the script and libraries not found.

Regards
--
Cyril SCETBON

On Mar 12, 2013, at 10:43 AM, 
cscetbon@orange.com wrote:

I'm already using Cassandra 1.2.2 with only one line to test the cassandra 
access :

rows = LOAD 'cassandra://twissandra/users' USING 
org.apache.cassandra.hadoop.pig.CassandraStorage();

extracted from the sample script provided in the sources
--
Cyril SCETBON

On Mar 12, 2013, at 6:57 AM, aaron morton 
mailto:aa...@thelastpickle.com>> wrote:

any idea why the function loadFunc does not work correctly ?
No sorry.
Not sure why you are linking to the CQL info or what Pig script / config you 
are running.
Did you follow the example in the examples/pig in the source distribution ?

Also please use at least cassandra 1.1.

Cheers

-
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 11/03/2013, at 9:39 AM, 
cscetbon@orange.com wrote:

You said all versions. However, when I try to access 
cassandra://twissandra/users based on 
http://www.datastax.com/docs/1.0/dml/using_cql I get :

2013-03-11 17:35:48,444 [main] INFO  org.apache.pig.Main - Apache Pig version 
0.11.0 (r1446324) compiled Feb 14 2013, 16:40:57
2013-03-11 17:35:48,445 [main] INFO  org.apache.pig.Main - Logging error 
messages to: /Users/cyril/pig_1363019748442.log
2013-03-11 17:35:48.583 java[13809:1203] Unable to load realm info from 
SCDynamicStore
2013-03-11 17:35:48,750 [main] INFO  org.apache.pig.impl.util.Utils - Default 
bootup file /Users/cyril/.pigbootup not found
2013-03-11 17:35:48,831 [main] INFO  
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to 
hadoop file system at: file:///
2013-03-11 17:35:49,235 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 
2245: Cannot get schema from loadFunc 
org.apache.cassandra.hadoop.pig.CassandraStorage

with pig 0.11.0

any idea why the function loadFunc does not work correctly ?

thanks
--
Cyril SCETBON

On Jan 18, 2013, at 7:00 PM, aaron morton 
mailto:aa...@thelastpickle.com>> wrote:

Silly question -- but does hive/pig hadoop etc work with cassandra
1.1.8?  Or only with 1.2?
all versions.

We are using astyanax library, which seems
to fail horribly on 1.2,
How does it fail ?
If you think you have a bug post it at https://github.com/Netflix/astyanax

Cheers

-
Aaron Morton
Freelance Cassandra Developer
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 18/01/2013, at 7:48 AM, James Lyons 
mailto:james.ly...@gmail.com>> wrote:

Silly question -- but does hive/pig hadoop etc work with cassandra
1.1.8?  Or only with 1.2?  We are using astyanax library, which seems
to fail horribly on 1.2, so we're still on 1.1.8.  But we're just
star

Re: Cassandra Compression and Wide Rows

[Sylvain Lebresne]

The way compression is implemented, it is oblivious to the CF being
wide-row or narrow-row. There is nothing intrinsically less efficient in
the compression for wide-rows.

--
Sylvain


On Fri, Mar 15, 2013 at 11:53 PM, Drew Kutcharian  wrote:

> Hey Guys,
>
> I remember reading somewhere that C* compression is not very effective
> when most of the CFs are in wide-row format and some folks turn the
> compression off and use disk level compression as a workaround. Considering
> that wide rows with composites are "first class citizens" in CQL3, is this
> still the case? Has there been any improvements on this?
>
> Thanks,
>
> Drew

Re: cql query not giving any result.

[Sylvain Lebresne]

CQL can't work correctly if 2 (CQL) columns have the same name. Now, to
allow upgrade from thrift, CQL does use some default names like "key" for
the Row key when there isn't anything else.

Honestly I think the easiest workaround here is probably to disambiguate
things manually. Typically, you could update the column family definition
to set the key_alias (in CfDef) to some name that make sense for you. This
will end up being the name of the Row key for CQL. You may also try issue a
RENAME from CQL to rename the row key, which "should" work. Typically
something like "ALTER KunderaExamples RENAME key TO rowKey".

--
Sylvain



On Sat, Mar 16, 2013 at 4:39 AM, Vivek Mishra  wrote:

> Any suggestions?
> -Vivek
>
> On Fri, Mar 15, 2013 at 5:20 PM, Vivek Mishra wrote:
>
>> Ok. So it's a case  when, CQL returns rowkey value as "key" and there is
>> also column present with name as "key".
>>
>> Sounds like a bug?
>>
>> -Vivek
>>
>>
>> On Fri, Mar 15, 2013 at 5:17 PM, Kuldeep Mishra > > wrote:
>>
>>> Hi Sylvain,
>>>   I created it using thrift client, here is column family creation
>>> script,
>>>
>>> Cassandra.Client client;
>>> CfDef user_Def = new CfDef();
>>> user_Def.name = "DOCTOR";
>>> user_Def.keyspace = "KunderaExamples";
>>> user_Def.setComparator_type("UTF8Type");
>>> user_Def.setDefault_validation_class("UTF8Type");
>>> user_Def.setKey_validation_class("UTF8Type");
>>> ColumnDef key = new ColumnDef(ByteBuffer.wrap("KEY".getBytes()),
>>> "UTF8Type");
>>> key.index_type = IndexType.KEYS;
>>> ColumnDef age = new ColumnDef(ByteBuffer.wrap("AGE".getBytes()),
>>> "UTF8Type");
>>> age.index_type = IndexType.KEYS;
>>> user_Def.addToColumn_metadata(key);
>>> user_Def.addToColumn_metadata(age);
>>>
>>> client.set_keyspace("KunderaExamples");
>>> client.system_add_column_family(user_Def);
>>>
>>>
>>> Thanks
>>> KK
>>>
>>>
>>> On Fri, Mar 15, 2013 at 4:24 PM, Sylvain Lebresne 
>>> wrote:
>>>
 On Fri, Mar 15, 2013 at 11:43 AM, Kuldeep Mishra <
 kuld.cs.mis...@gmail.com> wrote:

> Hi,
> Is it possible in Cassandra to make multiple column with same name ?,
> like in this particular scenario I have two column with same name as 
> "key",
> first one is rowkey and second on is column name .
>
>
 No, it shouldn't be possible and that is your problem. How did you
 created that table?

 --
 Sylvain


>
> Thanks and Regards
> Kuldeep
>
>
> On Fri, Mar 15, 2013 at 4:05 PM, Kuldeep Mishra <
> kuld.cs.mis...@gmail.com> wrote:
>
>>
>> Hi ,
>> Following cql query not returning any result
>> cqlsh:KunderaExamples> select * from "DOCTOR" where key='kuldeep';
>>
>>I have enabled secondary indexes on both column.
>>
>> Screen shot is attached
>>
>> Please help
>>
>>
>> --
>> Thanks and Regards
>> Kuldeep Kumar Mishra
>> +919540965199
>>
>
>
>
> --
> Thanks and Regards
> Kuldeep Kumar Mishra
> +919540965199
>


>>>
>>>
>>> --
>>> Thanks and Regards
>>> Kuldeep Kumar Mishra
>>> +919540965199
>>>
>>
>>
>

Re: cql query not giving any result.

[Vivek Mishra]

If this is the case, Why can't we restrict "key" as a keyword and not to be
used as a column name?

-Vivek

On Mon, Mar 18, 2013 at 2:37 PM, Sylvain Lebresne wrote:

> CQL can't work correctly if 2 (CQL) columns have the same name. Now, to
> allow upgrade from thrift, CQL does use some default names like "key" for
> the Row key when there isn't anything else.
>
> Honestly I think the easiest workaround here is probably to disambiguate
> things manually. Typically, you could update the column family definition
> to set the key_alias (in CfDef) to some name that make sense for you. This
> will end up being the name of the Row key for CQL. You may also try issue a
> RENAME from CQL to rename the row key, which "should" work. Typically
> something like "ALTER KunderaExamples RENAME key TO rowKey".
>
> --
> Sylvain
>
>
>
> On Sat, Mar 16, 2013 at 4:39 AM, Vivek Mishra wrote:
>
>> Any suggestions?
>> -Vivek
>>
>> On Fri, Mar 15, 2013 at 5:20 PM, Vivek Mishra wrote:
>>
>>> Ok. So it's a case  when, CQL returns rowkey value as "key" and there is
>>> also column present with name as "key".
>>>
>>> Sounds like a bug?
>>>
>>> -Vivek
>>>
>>>
>>> On Fri, Mar 15, 2013 at 5:17 PM, Kuldeep Mishra <
>>> kuld.cs.mis...@gmail.com> wrote:
>>>
 Hi Sylvain,
   I created it using thrift client, here is column family creation
 script,

 Cassandra.Client client;
 CfDef user_Def = new CfDef();
 user_Def.name = "DOCTOR";
 user_Def.keyspace = "KunderaExamples";
 user_Def.setComparator_type("UTF8Type");
 user_Def.setDefault_validation_class("UTF8Type");
 user_Def.setKey_validation_class("UTF8Type");
 ColumnDef key = new
 ColumnDef(ByteBuffer.wrap("KEY".getBytes()), "UTF8Type");
 key.index_type = IndexType.KEYS;
 ColumnDef age = new
 ColumnDef(ByteBuffer.wrap("AGE".getBytes()), "UTF8Type");
 age.index_type = IndexType.KEYS;
 user_Def.addToColumn_metadata(key);
 user_Def.addToColumn_metadata(age);

 client.set_keyspace("KunderaExamples");
 client.system_add_column_family(user_Def);


 Thanks
 KK


 On Fri, Mar 15, 2013 at 4:24 PM, Sylvain Lebresne >>> > wrote:

> On Fri, Mar 15, 2013 at 11:43 AM, Kuldeep Mishra <
> kuld.cs.mis...@gmail.com> wrote:
>
>> Hi,
>> Is it possible in Cassandra to make multiple column with same name ?,
>> like in this particular scenario I have two column with same name as 
>> "key",
>> first one is rowkey and second on is column name .
>>
>>
> No, it shouldn't be possible and that is your problem. How did you
> created that table?
>
> --
> Sylvain
>
>
>>
>> Thanks and Regards
>> Kuldeep
>>
>>
>> On Fri, Mar 15, 2013 at 4:05 PM, Kuldeep Mishra <
>> kuld.cs.mis...@gmail.com> wrote:
>>
>>>
>>> Hi ,
>>> Following cql query not returning any result
>>> cqlsh:KunderaExamples> select * from "DOCTOR" where
>>> key='kuldeep';
>>>
>>>I have enabled secondary indexes on both column.
>>>
>>> Screen shot is attached
>>>
>>> Please help
>>>
>>>
>>> --
>>> Thanks and Regards
>>> Kuldeep Kumar Mishra
>>> +919540965199
>>>
>>
>>
>>
>> --
>> Thanks and Regards
>> Kuldeep Kumar Mishra
>> +919540965199
>>
>
>


 --
 Thanks and Regards
 Kuldeep Kumar Mishra
 +919540965199

>>>
>>>
>>
>

Re: cql query not giving any result.

[Sylvain Lebresne]

> If this is the case, Why can't we restrict "key" as a keyword and not to
> be used as a column name?


This is only a problem when upgrading from thrift to CQL. Forbidding "key"
as a column name in thrift would be weird to say the least.
What could be done is that CQL could, when it picks the default name it
uses, pick one that is not used already. That's definitively possible
and please do open a JIRA ticket for that.

But at the end of the day, if you are going to use CQL, I highly suggest
picking meaningful names for your CQL columns, so you will want
to rename the default name that CQL picks for the row key initially.

--
Sylvain


>
> -Vivek
>
>
> On Mon, Mar 18, 2013 at 2:37 PM, Sylvain Lebresne wrote:
>
>> CQL can't work correctly if 2 (CQL) columns have the same name. Now, to
>> allow upgrade from thrift, CQL does use some default names like "key" for
>> the Row key when there isn't anything else.
>>
>> Honestly I think the easiest workaround here is probably to disambiguate
>> things manually. Typically, you could update the column family definition
>> to set the key_alias (in CfDef) to some name that make sense for you. This
>> will end up being the name of the Row key for CQL. You may also try issue a
>> RENAME from CQL to rename the row key, which "should" work. Typically
>> something like "ALTER KunderaExamples RENAME key TO rowKey".
>>
>> --
>> Sylvain
>>
>>
>>
>> On Sat, Mar 16, 2013 at 4:39 AM, Vivek Mishra wrote:
>>
>>> Any suggestions?
>>> -Vivek
>>>
>>> On Fri, Mar 15, 2013 at 5:20 PM, Vivek Mishra wrote:
>>>
 Ok. So it's a case  when, CQL returns rowkey value as "key" and there
 is also column present with name as "key".

 Sounds like a bug?

 -Vivek


 On Fri, Mar 15, 2013 at 5:17 PM, Kuldeep Mishra <
 kuld.cs.mis...@gmail.com> wrote:

> Hi Sylvain,
>   I created it using thrift client, here is column family creation
> script,
>
> Cassandra.Client client;
> CfDef user_Def = new CfDef();
> user_Def.name = "DOCTOR";
> user_Def.keyspace = "KunderaExamples";
> user_Def.setComparator_type("UTF8Type");
> user_Def.setDefault_validation_class("UTF8Type");
> user_Def.setKey_validation_class("UTF8Type");
> ColumnDef key = new
> ColumnDef(ByteBuffer.wrap("KEY".getBytes()), "UTF8Type");
> key.index_type = IndexType.KEYS;
> ColumnDef age = new
> ColumnDef(ByteBuffer.wrap("AGE".getBytes()), "UTF8Type");
> age.index_type = IndexType.KEYS;
> user_Def.addToColumn_metadata(key);
> user_Def.addToColumn_metadata(age);
>
> client.set_keyspace("KunderaExamples");
> client.system_add_column_family(user_Def);
>
>
> Thanks
> KK
>
>
> On Fri, Mar 15, 2013 at 4:24 PM, Sylvain Lebresne <
> sylv...@datastax.com> wrote:
>
>> On Fri, Mar 15, 2013 at 11:43 AM, Kuldeep Mishra <
>> kuld.cs.mis...@gmail.com> wrote:
>>
>>> Hi,
>>> Is it possible in Cassandra to make multiple column with same name
>>> ?, like in this particular scenario I have two column with same name as
>>> "key", first one is rowkey and second on is column name .
>>>
>>>
>> No, it shouldn't be possible and that is your problem. How did you
>> created that table?
>>
>> --
>> Sylvain
>>
>>
>>>
>>> Thanks and Regards
>>> Kuldeep
>>>
>>>
>>> On Fri, Mar 15, 2013 at 4:05 PM, Kuldeep Mishra <
>>> kuld.cs.mis...@gmail.com> wrote:
>>>

 Hi ,
 Following cql query not returning any result
 cqlsh:KunderaExamples> select * from "DOCTOR" where
 key='kuldeep';

I have enabled secondary indexes on both column.

 Screen shot is attached

 Please help


 --
 Thanks and Regards
 Kuldeep Kumar Mishra
 +919540965199

>>>
>>>
>>>
>>> --
>>> Thanks and Regards
>>> Kuldeep Kumar Mishra
>>> +919540965199
>>>
>>
>>
>
>
> --
> Thanks and Regards
> Kuldeep Kumar Mishra
> +919540965199
>


>>>
>>
>

[RELEASE] Apache Cassandra 1.2.3 released

[Sylvain Lebresne]

The Cassandra team is pleased to announce the release of Apache Cassandra
version 1.2.3.

Cassandra is a highly scalable second-generation distributed database,
bringing together Dynamo's fully distributed design and Bigtable's
ColumnFamily-based data model. You can read more here:

 http://cassandra.apache.org/

Downloads of source and binary distributions are listed in our download
section:

 http://cassandra.apache.org/download/

This version is a maintenance/bug fix release[1] on the 1.2 series. As
always,
please pay attention to the release notes[2] and Let us know[3] if you were
to
encounter any problem.

Enjoy!

[1]: http://goo.gl/jS1Nd (CHANGES.txt)
[2]: http://goo.gl/yyhwf (NEWS.txt)
[3]: https://issues.apache.org/jira/browse/CASSANDRA

Re: Cassandra Compression and Wide Rows

[Edward Capriolo]

Imho it is probably more efficient for wide. When you decompress 8k blocks
to get at a 200 byte row you create overhead , particularly young gen.
On Monday, March 18, 2013, Sylvain Lebresne  wrote:
> The way compression is implemented, it is oblivious to the CF being
wide-row or narrow-row. There is nothing intrinsically less efficient in
the compression for wide-rows.
> --
> Sylvain
>
> On Fri, Mar 15, 2013 at 11:53 PM, Drew Kutcharian  wrote:
>>
>> Hey Guys,
>>
>> I remember reading somewhere that C* compression is not very effective
when most of the CFs are in wide-row format and some folks turn the
compression off and use disk level compression as a workaround. Considering
that wide rows with composites are "first class citizens" in CQL3, is this
still the case? Has there been any improvements on this?
>>
>> Thanks,
>>
>> Drew
>

Unable to fetch large amount of rows

[Pushkar Prasad]

Hi,

 

I have following schema:

 

TimeStamp

MACAddress

Data Transfer

Data Rate

LocationID

 

PKEY is (TimeStamp, MACAddress). That means partitioning is on TimeStamp,
and data is ordered by MACAddress, and stored together physically (let me
know if my understanding is wrong). I have 1000 timestamps, and for each
timestamp, I have 500K different MACAddress. 

 

When I run the following query, I get RPC Timeout exceptions:

 

 

Select * from db_table where Timestamp='...'

 

>From my understanding, this should give all the rows with just one disk
seek, as all the records for a particular timeStamp. This should be very
quick, however, clearly, that doesn't seem to be the case. Is there
something I am missing here? Your help would be greatly appreciated.

 

Thanks

PP

Re: Backup solution

[Rene Kochen]

Hi Aaron,

Thank you for your answer!

My idea was to do the snapshots in the backup DC only. That way the backup
procedure will not affect the live DC. However I'm afraid that a
point-in-time recovery via the snapshots in the second DC (first restore
backup on backup DC and then repair live DC) will take too long. I expect
the data to grow significantly.

It makes more sense to use the second cluster as a hot standby (and make
snapshots on both clusters).

Rene

2013/3/16 Aaron Turner 

> On Fri, Mar 15, 2013 at 10:35 AM, Rene Kochen
>  wrote:
> > Hi Aaron,
> >
> > We have many deployments, but typically:
> >
> > - Live cluster of six nodes, replication factor = 3.
> > - A node processes more reads than writes (approximately 100 get_slices
> > per/second, narrow rows).
> > - Data per node is about 50 to 100 GBytes.
> > - We should recover within 4 hours.
> >
> > The idea is to put the backup cluster close to the live cluster with a
> > gigabit connection only for Cassandra.
>
> 100 reads/sec/node doesn't sound like a lot to me... And 100G/node is
> far below the recommended limit.  Sounds to me  you've possibly over
> spec'd your cluster (not a bad thing, just an observation).  Of
> course, if your data set is growing, then...
>
> That said, I wouldn't consider a single node in a 2nd DC receiving
> updates via Cassandra a "backup".  That's because a bug in cassandra
> which corrupts your data or a user accidentally doing the wrong thing
> (like issuing deletes they shouldn't) means that get's replicated to
> all your nodes- including the one in the other DC.
>
> A real backup would be to take snapshots on the nodes and then copy
> them off the cluster.
>
> I'd say replication is good if you want a hot-standby for a disaster
> recovery site so you can quickly recover from a hardware fault.
> Especially if you have a 4hr SLA, how are you going to get your
> primary DC back up after a fire, earthquake, etc in 4 hours?  Heck, a
> switch failure might knock you out for 4 hours depending on how
> quickly you can swap another one in and how recent your config backups
> are.
>
> Better to have a DR site with a smaller set of nodes with the data
> ready to go.  Maybe they won't be as fast, but hopefully you can make
> sure the most important queries are handled.  But for that, I would
> probably go with something more then just a single node in the DR DC.
>
> One thing to remember is that compactions will impact the feasible
> single node size to something smaller then you can potentially
> allocate disk space for.   Ie: just because you can build a 4TB disk
> array, doesn't mean you can have a single Cassandra node with 4TB of
> data.  Typically, people around here seem to recommend ~400GB, but
> that depends on hardware.
>
> Honestly, for the price of a single computer you could test this
> pretty easy.  That's what I'd do.
>
> --
> Aaron Turner
> http://synfin.net/ Twitter: @synfinatic
> http://tcpreplay.synfin.net/ - Pcap editing and replay tools for Unix &
> Windows
> Those who would give up essential Liberty, to purchase a little temporary
> Safety, deserve neither Liberty nor Safety.
> -- Benjamin Franklin
> "carpe diem quam minimum credula postero"
>

Unable to fetch large amount of rows

[Pushkar Prasad]

Hi,

 

I have following schema:

 

TimeStamp

MACAddress

Data Transfer

Data Rate

LocationID

 

PKEY is (TimeStamp, MACAddress). That means partitioning is on TimeStamp,
and data is ordered by MACAddress, and stored together physically (let me
know if my understanding is wrong). I have 1000 timestamps, and for each
timestamp, I have 500K different MACAddress. 

 

When I run the following query, I get RPC Timeout exceptions:

 

 

Select * from db_table where Timestamp='...'

 

>From my understanding, this should give all the rows with just one disk
seek, as all the records for a particular timeStamp. This should be very
quick, however, clearly, that doesn't seem to be the case. Is there
something I am missing here? Your help would be greatly appreciated.

 

Thanks

PP

Re: Backup solution

[Aaron Turner]

On Mon, Mar 18, 2013 at 7:33 AM, Rene Kochen
 wrote:
> Hi Aaron,
>
> Thank you for your answer!
>
> My idea was to do the snapshots in the backup DC only. That way the backup
> procedure will not affect the live DC. However I'm afraid that a
> point-in-time recovery via the snapshots in the second DC (first restore
> backup on backup DC and then repair live DC) will take too long. I expect
> the data to grow significantly.
>
> It makes more sense to use the second cluster as a hot standby (and make
> snapshots on both clusters).

Remember, snapshots are *cheap*.  There's almost literally zero I/O
associated with a snapshot.  Backing up all that data off the system
is a different story, but at least it's large sequential reads which
is pretty well optimized.

-- 
Aaron Turner
http://synfin.net/ Twitter: @synfinatic
http://tcpreplay.synfin.net/ - Pcap editing and replay tools for Unix & Windows
Those who would give up essential Liberty, to purchase a little temporary
Safety, deserve neither Liberty nor Safety.
-- Benjamin Franklin
"carpe diem quam minimum credula postero"

Re: 33million hinted handoffs from nowhere

[aaron morton]

You can check which nodes hints are being held for using the JMX api. Look for 
the org.apache.cassandra.db:type=HintedHandoffManager MBean and call the 
listEndpointsPendingHints() function. 

There are two points where hints may be stored, if the node is down when the 
request started or if the node timed out and did not return before rpc_timeout. 
To check for the first, look for log lines about a node being "dead" on the 
coordinator. To check for the second look for dropped messages on the other 
nodes. This will be logged, or you can use nodetool tpstats to look for them.

Cheers
  
-
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 15/03/2013, at 2:30 AM, Andras Szerdahelyi 
 wrote:

> ( The previous letter was sent prematurely, sorry. )
> 
> This node is the only node being written to, but the Cfs being written 
> replicate to almost all of the other nodes
> My understanding is that hinted handoff is mutations kept around on the 
> coordinator node, to be replayed when the target node re-appears on the ring. 
> All my nodes are up and again, no hinted handoff is logged on the node itself
> 
> Thanks!
> Andras
> 
> From: Andras Szerdahelyi 
> Date: Thursday 14 March 2013 14:25
> To: "user@cassandra.apache.org" 
> Subject: 33million hinted handoffs from nowhere
> 
> Hi list,
> 
> I am experiencing seemingly uncontrollable and unexplained growth of my 
> HintedHandoff CF on a single node. Unexplained because there are no hinted 
> handoffs being logged on the node, uncontrollable because I see 33 million 
> inserts in cfstats and the size of the stables is over 10 gigs all in an hour 
> of uptime. 
> 
> 
> I have done the following to try and reproduce this:
> 
> - shut down my cluster
> - on all nodes: remove sstables from the HintsColumnFamily data dir
> - on all nodes: remove commit logs
> - start all nodes but the one that’s showing this problem
> - nothing is writing to any of the nodes. There are no hinted handoff going 
> on anywhere
> - bring back the node in question last
> - few seconds after boot:
> 
> Column Family: HintsColumnFamily
> SSTable count: 1
> Space used (live): 44946532
> Space used (total): 44946532
> Number of Keys (estimate): 256
> Memtable Columns Count: 17840
> Memtable Data Size: 17569909
> Memtable Switch Count: 2
> Read Count: 0
> Read Latency: NaN ms.
> Write Count: 184836
> Write Latency: 0.668 ms.
> Pending Tasks: 0
> Bloom Filter False Postives: 0
> Bloom Filter False Ratio: 0.0
> Bloom Filter Space Used: 16
> Compacted row minimum size: 20924301
> Compacted row maximum size: 25109160
> Compacted row mean size: 25109160
> 
> 
> 
> 

Re: About the heap

[aaron morton]

> If you have the chance, could you expand on m1.xlarge being the much better 
> choice? 
For background 
http://perfcap.blogspot.co.nz/2011/03/understanding-and-using-amazon-ebs.html

it's an older post, but the idea is the same. On smaller machines you share 
more, and sharing is bad. Many times I've seen clusters moved from m1.xlarge 
have better performance and less issues, such as nodes flapping. 

C* runs a lot of threads, running these on a system with 2 cores is less 
efficient that 4 and 4 is less efficient than 8. Memory wise a 4Gb to 8Gb JVM 
heap is desirable, as is as much memory as possible for the OS page cache or C* 
row cache. So a machine with 16GB and 4 to 8 cores is a good starting point. 

> of expanding from a 12 node -> 24 node cluster using .large instances, vs. 
> upgrading all instances to m1.xlarge, soon and the justifications would be 
> helpful (although Aaron says so does help ;) ).  
Aside from the CPU / Memory / IO issues, you will also need to consider the 
data load per node. In EC2 try to keep between 300GB and 500GB per node. Once 
you get above 500GB maintenance tasks like repair and replacing nodes take a 
long time. 

> Doubling the cluster size would, in theory, halve the time for this overhead, 
> but would still impact performance during that time.  Going to xlarge would 
> lessen the impact of these activities on operations.
My experience is that an m1.xlarge is more than twice the performance of an 
m1.large. 

Hope that helps. 

-
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 15/03/2013, at 4:44 AM, Michael Theroux  wrote:

> Hi Aaron,
> 
> If you have the chance, could you expand on m1.xlarge being the much better 
> choice?  We are going to need to make a choice of expanding from a 12 node -> 
> 24 node cluster using .large instances, vs. upgrading all instances to 
> m1.xlarge, soon and the justifications would be helpful (although Aaron says 
> so does help ;) ).  
> 
> One obvious reason is administrating a 24 node cluster does add person-time 
> overhead.  
> 
> Another reason includes less impact of maintenance activities such as repair, 
> as these activites have significant CPU overhead.  Doubling the cluster size 
> would, in theory, halve the time for this overhead, but would still impact 
> performance during that time.  Going to xlarge would lessen the impact of 
> these activities on operations.
> 
> Anything else?
> 
> Thanks,
> 
> -Mike
> 
> On Mar 14, 2013, at 9:27 AM, aaron morton wrote:
> 
>>> Because of this I have an unstable cluster and have no other choice than 
>>> use Amazon EC2 xLarge instances when we would rather use twice more EC2 
>>> Large nodes.
>> m1.xlarge is a MUCH better choice than m1.large.
>> You get more ram and better IO and less steal. Using half as many m1.xlarge 
>> is the way to go. 
>> 
>>> My heap is actually changing from 3-4 GB to 6 GB and sometimes growing to 
>>> the max 8 GB (crashing the node).
>> How is it crashing ?
>> Are you getting too much GC or running OOM ? 
>> Are you using the default GC configuration ?
>> Is cassandra logging a lot of GC warnings ?
>> 
>> If you are running OOM then something has to change. Maybe bloom filters, 
>> maybe caches.
>> 
>> Enable the GC logging in cassandra-env.sh to check how low a CMS compaction 
>> get's the heap, or use some other tool. That will give an idea of how much 
>> memory you are using. 
>> 
>> Here is some background on what is kept on heap in pre 1.2
>> http://www.mail-archive.com/user@cassandra.apache.org/msg25762.html
>> 
>> Cheers
>> 
>> -
>> Aaron Morton
>> Freelance Cassandra Consultant
>> New Zealand
>> 
>> @aaronmorton
>> http://www.thelastpickle.com
>> 
>> On 13/03/2013, at 12:19 PM, Wei Zhu  wrote:
>> 
>>> Here is the JIRA I submitted regarding the ancestor.
>>> 
>>> https://issues.apache.org/jira/browse/CASSANDRA-5342
>>> 
>>> -Wei
>>> 
>>> 
>>> - Original Message -
>>> From: "Wei Zhu" 
>>> To: user@cassandra.apache.org
>>> Sent: Wednesday, March 13, 2013 11:35:29 AM
>>> Subject: Re: About the heap
>>> 
>>> Hi Dean,
>>> The index_interval is controlling the sampling of the SSTable to speed up 
>>> the lookup of the keys in the SSTable. Here is the code:
>>> 
>>> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/DataTracker.java#L478
>>> 
>>> To increase the interval meaning, taking less samples, less memory, slower 
>>> lookup for read.
>>> 
>>> I did do a heap dump on my production system which caused about 10 seconds 
>>> pause of the node. I found something interesting, for LCS, it could involve 
>>> thousands of SSTables for one compaction, the ancestors are recorded in 
>>> case something goes wrong during the compaction. But those are never 
>>> removed after the compaction is done. In our case, it takes about 1G of 
>>> heap memory to store that. I am going to submit a JIRA for that. 
>>> 
>>> Here is the culprit

Re: Iterating over all keys in column family with LongType key

[aaron morton]

> This strategy, however, doesn't seem to work with LongFamily.  I've tried
> various combinations of key ranges (0, 0), (1, 0), (0, Long.MaxValue), but
> none iterate over the entire family.  Any ideas?
What results do you see?
Have you tried Long.MinValue to Long.MaxValue ? 

If you are just starting out consider using this client, it's getting a lot of 
active development https://github.com/Netflix/astyanax

Finally, we are trying to direct client specific questions to the client 
mailing list. Please ask there next time. 

Cheers

-
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 15/03/2013, at 8:17 AM, sj.climber  wrote:

> Hi,
> 
> I have two column families, one keyed by UTF8Type (let's call it
> StringFamily), the other by LongType (LongFamily).  Both are stored in a
> keyspace using the RandomPartitioner.  I need to iterate over all columns in
> both families, although order does not matter.
> 
> Iterating over StringFamily is easy.  I'm currently using Pelops.  I request
> pages of 1000 results, with the first key slice specifying the following:
> Selector.newKeyRange("", "", PAGE_SIZE);
> For every page thereafter, I replace the first argument with the key of the
> last item returned in the previous page.
> 
> This strategy, however, doesn't seem to work with LongFamily.  I've tried
> various combinations of key ranges (0, 0), (1, 0), (0, Long.MaxValue), but
> none iterate over the entire family.  Any ideas?
> 
> Thanks in advance for your help!
> 
> 
> 
> --
> View this message in context: 
> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Iterating-over-all-keys-in-column-family-with-LongType-key-tp7586373.html
> Sent from the cassandra-u...@incubator.apache.org mailing list archive at 
> Nabble.com.

Re: Cassandra Platform Support

[aaron morton]

> What are the supported platforms for Cassandra?
Linux is the most used, there are people using Windows. 

> we have run into issues with some of the native calls used by Cassandra.
Could you provide some more information and/or file a ticket on 
https://issues.apache.org/jira/browse/CASSANDRA

You are somewhat exploring new territory, if you can ask raise some specific 
issues on the user list or in tickets you'll probably get the help you need. 
And the project will be all the better. 

Thanks
 
-
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 15/03/2013, at 9:31 AM, Rick Cole  wrote:

> We are looking to use Cassandra for our persistent store. However, we have 
> requirements to support multiple platforms including Windows Server, Linux, 
> HP-UX, AIX and Solaris. I have searched the web and literally find nothing 
> about Cassandra running on some of these platforms. I don't even find posts 
> asking this question. What are the supported platforms for Cassandra? I see 
> DataStax provides distros for Windows, Mac and Linux. On other platforms 
> (namely HP-UX), we have run into issues with some of the native calls used by 
> Cassandra. We have been able to successfully recompile the native Snappy 
> libraries on HP-UX (about to try other platforms), however it looks like 
> there are multiple sun.misc.Unsafe calls made by Cassandra when doing other 
> operations (ie: comparing arrays). Has anyone been successful in getting 
> Cassandra up and running on any or all of these platforms? Thank you for any 
> help.
>  
> Rick

Re: repair hangs

[aaron morton]

> /raid0/cassandra/data/OpsCenter/events_timeline/OpsCenter-events_timeline-hf-1-Data.db
> is not compatible with current version ib
> --
This can be fixed with a nodetool upgradesstables

Cheers

-
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 15/03/2013, at 2:20 PM, Dane Miller  wrote:

> On Thu, Mar 14, 2013 at 6:34 AM, aaron morton  wrote:
>>> 1. is this a nodetool bug?  is there any way to propagate the
>>> java.io.IOException back to nodetool?
>> The repair continues to work even if nodetool fails, it's a server side 
>> thing.
>> 
>>> 2. network problems on EC2, I'm shocked!  are there recommended
>>> network settings for EC2?
>> Streaming does not put a timeout on the socket, in this case check the 
>> 10.82.233.59
>> node to see why the pipe broke.
> 
> Good catch.  There are indeed errors on node 10.82.233.59:
> 
> cassandra3:/var/log/cassandra# grep -A1 ERROR system.log
> ERROR [Thread-5571] 2013-03-11 23:38:12,554 CassandraDaemon.java (line
> 132) Exception in thread Thread[Thread-5571,5,main]
> java.lang.UnsupportedOperationException: SSTable
> /raid0/cassandra/data/OpsCenter/events_timeline/OpsCenter-events_timeline-hf-1-Data.db
> is not compatible with current version ib
> --
> ERROR [Thread-5572] 2013-03-11 23:38:12,692 CassandraDaemon.java (line
> 132) Exception in thread Thread[Thread-5572,5,main]
> java.lang.UnsupportedOperationException: SSTable
> /raid0/cassandra/data/OpsCenter/rollups60/OpsCenter-rollups60-hf-766-Data.db
> is not compatible with current version ib
> --
> ERROR [Thread-5612] 2013-03-11 23:39:55,931 CassandraDaemon.java (line
> 132) Exception in thread Thread[Thread-5612,5,main]
> java.lang.UnsupportedOperationException: SSTable
> /raid0/cassandra/data/OpsCenter/rollups300/OpsCenter-rollups300-hf-506-Data.db
> is not compatible with current version ib
> --
> 
> 
> This attempt to create a new cluster from backups of the original
> cluster has not been going well.  These errors lead me to believe I
> need to focus on repairing the original cluster first.
> 
> Dane

Re: secondary index problem

[aaron morton]

Brett, 
Do you have some steps to reproduce the problem ? If so please create a 
ticket on jira. 

Cheers

-
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 16/03/2013, at 11:40 AM, Janne Jalkanen  wrote:

> 
> This could be either of the following bugs (which might be the same thing).  
> I get it too every time I recycle a node on 1.1.10.
> 
> https://issues.apache.org/jira/browse/CASSANDRA-4973
> or
> https://issues.apache.org/jira/browse/CASSANDRA-4785
> 
> /Janne
> 
> On Mar 15, 2013, at 23:24 , Brett Tinling  wrote:
> 
>> We have a CF with an indexed column 'type', but we get incomplete results 
>> when we query that CF for all rows matching 'type'.  We can find the missing 
>> rows if we query by key.
>> 
>> * we are seeing this on a small, single node, 1.2.2 instance with few rows.
>> * we use thrift execute_cql_query, no CL is specified
>> * none of repair, restart, compact, scrub helped
>> 
>> Finally, nodetool rebuild_index fixed it.  
>> 
>> Is index rebuild something we need to do periodically?  How often?  Is there 
>> a way to know when it needs to be done?  Do we have to run rebuild on all 
>> nodes?
>> 
>> We have not noticed this until 1.2
>> 
>> Regards,
>>  - Brett
>> 
>> 
>> 
>> 
>> 
>> 
> 

Re: Waiting on read repair?

[aaron morton]

> 1. With a ConsistencyLevel of quorum, does
> FBUtilities.waitForFutures() wait for read repair to complete before
> returning?
No
That's just a utility method. 
Nothing on the read path waits for Read Repair, and controlled by 
read_repair_chance CF property, it's all async to the client request. 
There is no CL, the messages are sent to individual nodes. 

> 2. When read repair applies a mutation, it needs to obtain a lock for
> the associated memtable.
What lock are you referring to?
When Read Repair (the RowDataResolver) wants to send a mutation it uses the 
MessageServer. On the write path there is a server wide RW lock call the sync 
lock.

> I've seen readrepair spend a few seconds stalling in 
> org.apache.cassandra.db.Table.apply).
This  could be contention around the sync lock, look for blocked tasks in the 
flush writer thread pool. 

I did a talk on cassandra internals at Apache Con 3 weeks ago, not sure when 
the video is going to be up but here are the slides 
http://www.slideshare.net/aaronmorton/apachecon-nafeb2013

Cheers

-
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 16/03/2013, at 12:21 PM, Jasdeep Hundal  wrote:

> I've got a couple of questions related issues I'm encountering using
> Cassandra under a heavy write load:
> 
> 1. With a ConsistencyLevel of quorum, does
> FBUtilities.waitForFutures() wait for read repair to complete before
> returning?
> 2. When read repair applies a mutation, it needs to obtain a lock for
> the associated memtable. Does compaction obtain this same lock? (I'm
> asking because I've seen readrepair spend a few seconds stalling in
> org.apache.cassandra.db.Table.apply).
> 
> Thanks,
> Jasdeep

Re: Can't replace dead node

[aaron morton]

If a node is a seed node it will not bootstrap data from others the first time 
it starts. 

You can always run a nodetool repair when you think data is not where it should 
be. 

Cheers

-
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 16/03/2013, at 1:16 PM, Andrey Ilinykh  wrote:

> More info.
> I see this problem only if rest of cluster runs 1.1.5 and I try to replace 
> dead node by version 1.1.7 or higher. If I upgrade rest of cluster to the 
> same version (I tried 1.1.10) everything is ok. I guess there is some 
> incompatibility between 1.1.5 and 1.1.7 and higher.
> 
> Thank you,
>   Andrey
> 
> 
> On Fri, Mar 15, 2013 at 11:39 AM, Andrey Ilinykh  wrote:
> I removed Priam and get the same picture. 
> 
> 
> What I do is- I added to cassandra-env.sh two lines and start cassandra.
> 
> JVM_OPTS="$JVM_OPTS 
> -Dcassandra.initial_token=aaba"
> JVM_OPTS="$JVM_OPTS 
> -Dcassandra.replace_token=aaba"
> 
> Then I can successfully run ring command
> 
> 
> Note: Ownership information does not include topology, please specify a 
> keyspace. 
> Address DC  RackStatus State   LoadOwns   
>  Token   
>   
>  Token(bytes[aaba])
> 10.28.241.14us-east 1a  Up Normal  251.96 GB   33.33% 
>  Token(bytes[0010])
> 10.240.119.230  us-east 1b  Up Normal  252.48 GB   33.33% 
>  Token(bytes[5565])
> 10.147.174.27   us-east 1c  Up Normal  11.26 KB33.33% 
>  Token(bytes[aaba])
> 
> It shows the current node as part of the ring, but it is empty. In data 
> directory I can see only system key space.
> 
> There is no any errors in log file. It just doen't stream data from other 
> nodes. 
> I can launch 1.1.6 but not 1.1.7 or higher. Any ideas what was changed in 
> 1.1.7?
> 
> Thank you,
>   Andrey
> 
> 
> 
> INFO [main] 2013-03-15 18:20:45,303 AbstractCassandraDaemon.java (line 101) 
> Logging initialized
>  INFO [main] 2013-03-15 18:20:45,309 AbstractCassandraDaemon.java (line 122) 
> JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.6.0_35
>  INFO [main] 2013-03-15 18:20:45,310 AbstractCassandraDaemon.java (line 123) 
> Heap size: 1931476992/1931476992
>  INFO [main] 2013-03-15 18:20:45,311 AbstractCassandraDaemon.java (line 124) 
> Classpath: 
> /opt/apache-cassandra-1.1.10/bin/../conf:/opt/apache-cassandra-1.1.10/bin/../build/classes/main:/opt/apache-cassandra-1.1.10/bin/../build/classes/thrift:/opt/apache-cassandra-1.1.10/bin/../lib/antlr-3.2.jar:/opt/apache-cassandra-1.1.10/bin/../lib/apache-cassandra-1.1.10.jar:/opt/apache-cassandra-1.1.10/bin/../lib/apache-cassandra-clientutil-1.1.10.jar:/opt/apache-cassandra-1.1.10/bin/../lib/apache-cassandra-thrift-1.1.10.jar:/opt/apache-cassandra-1.1.10/bin/../lib/avro-1.4.0-fixes.jar:/opt/apache-cassandra-1.1.10/bin/../lib/avro-1.4.0-sources-fixes.jar:/opt/apache-cassandra-1.1.10/bin/../lib/commons-cli-1.1.jar:/opt/apache-cassandra-1.1.10/bin/../lib/commons-codec-1.2.jar:/opt/apache-cassandra-1.1.10/bin/../lib/commons-lang-2.4.jar:/opt/apache-cassandra-1.1.10/bin/../lib/compress-lzf-0.8.4.jar:/opt/apache-cassandra-1.1.10/bin/../lib/concurrentlinkedhashmap-lru-1.3.jar:/opt/apache-cassandra-1.1.10/bin/../lib/guava-r08.jar:/opt/apache-cassandra-1.1.10/bin/../lib/high-scale-lib-1.1.2.jar:/opt/apache-cassandra-1.1.10/bin/../lib/jackson-core-asl-1.9.2.jar:/opt/apache-cassandra-1.1.10/bin/../lib/jackson-mapper-asl-1.9.2.jar:/opt/apache-cassandra-1.1.10/bin/../lib/jamm-0.2.5.jar:/opt/apache-cassandra-1.1.10/bin/../lib/jline-0.9.94.jar:/opt/apache-cassandra-1.1.10/bin/../lib/jna.jar:/opt/apache-cassandra-1.1.10/bin/../lib/json-simple-1.1.jar:/opt/apache-cassandra-1.1.10/bin/../lib/libthrift-0.7.0.jar:/opt/apache-cassandra-1.1.10/bin/../lib/log4j-1.2.16.jar:/opt/apache-cassandra-1.1.10/bin/../lib/metrics-core-2.0.3.jar:/opt/apache-cassandra-1.1.10/bin/../lib/priam.jar:/opt/apache-cassandra-1.1.10/bin/../lib/servlet-api-2.5-20081211.jar:/opt/apache-cassandra-1.1.10/bin/../lib/slf4j-api-1.6.1.jar:/opt/apache-cassandra-1.1.10/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/apache-cassandra-1.1.10/bin/../lib/snakeyaml-1.6.jar:/opt/apache-cassandra-1.1.10/bin/../lib/snappy-java-1.0.4.1.jar:/opt/apache-cassandra-1.1.10/bin/../lib/snaptree-0.1.jar:/opt/apache-cassandra-1.1.10/bin/../lib/jamm-0.2.5.jar
>  INFO [main] 2013-03-15 18:20:47,406 CLibrary.java (line 111) JNA mlockall 
> successful
>  INFO [main] 2013-03-15 18:20:47,419 DatabaseDescriptor.java (line 123) 
> Loading settings from file:/opt/apache-cassandra-1.1.10/conf/cassandra.yaml
>  INFO [main] 2013-03-15 18:20:47,840 DatabaseDescrip

Re: Lots of Deleted Rows Came back after upgrade 1.1.6 to 1.1.10

[aaron morton]

> As you see, this node thinks lots of ranges are out of sync which shouldn't 
> be the case as successful repairs where done every night prior to the 
> upgrade. 
Could this be explained by writes occurring during the upgrade process ? 

> I found this bug which touches timestamp and tomstones which was fixed in 
> 1.1.10 but am not 100% sure if it could be related to this issue: 
> https://issues.apache.org/jira/browse/CASSANDRA-5153
Me neither, but the issue was fixed in 1.1.0

>  It appears that the repair task that I executed after upgrade, brought back 
> lots of deleted rows into life.
Was it entire rows or columns in a row?
Do you know if row level or column level deletes were used ? 

Can you look at the data in cassanca-cli and confirm the timestamps on the 
columns make sense ?  

Cheers

-
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 16/03/2013, at 2:31 PM, Arya Goudarzi  wrote:

> Hi,
> 
> I have upgraded our test cluster from 1.1.6 to 1.1.10. Followed by running 
> repairs. It appears that the repair task that I executed after upgrade, 
> brought back lots of deleted rows into life. Here are some logistics:
> 
> - The upgraded cluster started from 1.1.1 -> 1.1.2 -> 1.1.5 -> 1.1.6 
> - Old cluster: 4 node, C* 1.1.6 with RF3 using NetworkTopology;
> - Upgrade to : 1.1.10 with all other settings the same;
> - Successful repairs were being done on this cluster every night;
> - Our clients use nanosecond precision timestamp for cassandra calls;
> - After upgrade, while running repair I say some log messages like this in 
> one node:
> 
> system.log.5: INFO [AntiEntropyStage:1] 2013-03-15 19:55:54,847 
> AntiEntropyService.java (line 1022) [repair 
> #0990f320-8da9-11e2--e9b2bd8ea1bd] Endpoints /XX.194.60 and /23.20.207.56 
> have 2223 range(s) out of sync for App
> system.log.5: INFO [AntiEntropyStage:1] 2013-03-15 19:55:54,877 
> AntiEntropyService.java (line 1022) [repair 
> #0990f320-8da9-11e2--e9b2bd8ea1bd] Endpoints /XX.250.43 and /23.20.207.56 
> have 161 range(s) out of sync for App
> system.log.5: INFO [AntiEntropyStage:1] 2013-03-15 19:55:55,097 
> AntiEntropyService.java (line 1022) [repair 
> #0990f320-8da9-11e2--e9b2bd8ea1bd] Endpoints /XX.194.60 and /23.20.250.43 
> have 2294 range(s) out of sync for App
> system.log.5: INFO [AntiEntropyStage:1] 2013-03-15 19:55:59,190 
> AntiEntropyService.java (line 789) [repair 
> #0990f320-8da9-11e2--e9b2bd8ea1bd] App is fully synced (13 remaining 
> column family to sync for this session)
> 
> As you see, this node thinks lots of ranges are out of sync which shouldn't 
> be the case as successful repairs where done every night prior to the 
> upgrade. 
> 
> The App CF uses SizeTiered with gc_grace of 10 days. It has caching = 'ALL', 
> and it is fairly small (11Mb on each node).
> 
> I found this bug which touches timestamp and tomstones which was fixed in 
> 1.1.10 but am not 100% sure if it could be related to this issue: 
> https://issues.apache.org/jira/browse/CASSANDRA-5153
> 
> Any advice on how to dig deeper into this would be appreciated.
> 
> Thanks,
> -Arya
> 
> 

Re: Lots of Deleted Rows Came back after upgrade 1.1.6 to 1.1.10

[Arya Goudarzi]

Hi Aaron:

Thanks for your attention.

The cluster in question is a 4 node sandbox cluster we have that does not
have much traffic. I was able to chase down this issue on a CF that doesn't
change much.

That bug was flagged as fixed on 1.1.10.

They were row level deletes.

We use the nanosecond precision so they are something like this
: 1363379219546536704. Although this is recent for the last friday. I have
to find one with timestamp which is old and came back to life. Doing this
investigation this week and once I collect more info and reproduce from
snapshot, I'll let you know.

Cheers,
-Arya



On Mon, Mar 18, 2013 at 10:45 AM, aaron morton wrote:

> As you see, this node thinks lots of ranges are out of sync which
> shouldn't be the case as successful repairs where done every night prior to
> the upgrade.
>
> Could this be explained by writes occurring during the upgrade process ?
>
> I found this bug which touches timestamp and tomstones which was fixed in
> 1.1.10 but am not 100% sure if it could be related to this issue:
> https://issues.apache.org/jira/browse/CASSANDRA-5153
>
> Me neither, but the issue was fixed in 1.1.0
>
>  It appears that the repair task that I executed after upgrade, brought
> back lots of deleted rows into life.
>
> Was it entire rows or columns in a row?
> Do you know if row level or column level deletes were used ?
>
> Can you look at the data in cassanca-cli and confirm the timestamps on the
> columns make sense ?
>
> Cheers
>
> -
> Aaron Morton
> Freelance Cassandra Consultant
> New Zealand
>
> @aaronmorton
> http://www.thelastpickle.com
>
> On 16/03/2013, at 2:31 PM, Arya Goudarzi  wrote:
>
> Hi,
>
> I have upgraded our test cluster from 1.1.6 to 1.1.10. Followed by running
> repairs. It appears that the repair task that I executed after upgrade,
> brought back lots of deleted rows into life. Here are some logistics:
>
> - The upgraded cluster started from 1.1.1 -> 1.1.2 -> 1.1.5 -> 1.1.6
> - Old cluster: 4 node, C* 1.1.6 with RF3 using NetworkTopology;
> - Upgrade to : 1.1.10 with all other settings the same;
> - Successful repairs were being done on this cluster every night;
> - Our clients use nanosecond precision timestamp for cassandra calls;
> - After upgrade, while running repair I say some log messages like this in
> one node:
>
> system.log.5: INFO [AntiEntropyStage:1] 2013-03-15 19:55:54,847
> AntiEntropyService.java (line 1022) [repair
> #0990f320-8da9-11e2--e9b2bd8ea1bd] Endpoints /XX.194.60 and /
> 23.20.207.56 have 2223 range(s) out of sync for App
> system.log.5: INFO [AntiEntropyStage:1] 2013-03-15 19:55:54,877
> AntiEntropyService.java (line 1022) [repair
> #0990f320-8da9-11e2--e9b2bd8ea1bd] Endpoints /XX.250.43 and /
> 23.20.207.56 have 161 range(s) out of sync for App
> system.log.5: INFO [AntiEntropyStage:1] 2013-03-15 19:55:55,097
> AntiEntropyService.java (line 1022) [repair
> #0990f320-8da9-11e2--e9b2bd8ea1bd] Endpoints /XX.194.60 and /
> 23.20.250.43 have 2294 range(s) out of sync for App
> system.log.5: INFO [AntiEntropyStage:1] 2013-03-15 19:55:59,190
> AntiEntropyService.java (line 789) [repair
> #0990f320-8da9-11e2--e9b2bd8ea1bd] App is fully synced (13 remaining
> column family to sync for this session)
>
> As you see, this node thinks lots of ranges are out of sync which
> shouldn't be the case as successful repairs where done every night prior to
> the upgrade.
>
> The App CF uses SizeTiered with gc_grace of 10 days. It has caching =
> 'ALL', and it is fairly small (11Mb on each node).
>
> I found this bug which touches timestamp and tomstones which was fixed in
> 1.1.10 but am not 100% sure if it could be related to this issue:
> https://issues.apache.org/jira/browse/CASSANDRA-5153
>
> Any advice on how to dig deeper into this would be appreciated.
>
> Thanks,
> -Arya
>
>
>
>

Errors on replica nodes halt repair

[Dane Miller]

I'm having trouble completing a repair on several of my nodes due to
errors during compaction.  This is a 6 node cluster using the simple
replication strategy, rf=3, with each node assigned a single token.
I'm running "nodetool repair -pr" on node1, which progresses until a
specific keyspace then appears to hang.  On the replicas, nodes 2 and
3, I  find errors that seem related to compaction.  I'm considering
running a scrub if I can determine the column family where the errors
occur.  But I'm not confident I understand the problem, and I'm wary
of making it worse. What's the best way to recover from these errors?
Note, this cluster was recently upgraded from 1.1.6 to 1.2.1, then to
1.2.2.


node2

ERROR [Thread-97275] 2013-03-13 23:51:30,359 CassandraDaemon.java
(line 133) Exception in thread Thread[Thread-97275,5,main]
java.lang.RuntimeException: Last written key
DecoratedKey(161894077670705622023702574770140080251,
757365723a3a313a3a373537363636393130) >= current key DecoratedKey(

ERROR [CompactionExecutor:7697] 2013-03-15 21:45:59,584
CassandraDaemon.java (line 133) Exception in thread Thread[Compactio
nExecutor:7697,1,main]
java.lang.AssertionError: originally calculated column size of
321455446 but now it is 321455483


node3

ERROR [Thread-97525] 2013-03-13 23:51:44,788 CassandraDaemon.java
(line 133) Exception in thread Thread[Thread-97525,5,main]
java.lang.RuntimeException: Last written key
DecoratedKey(161894077670705622023702574770140080251,
757365723a3a313a3a373537363636393130) >= current key DecoratedKey

ERROR [Thread-97564] 2013-03-13 23:54:03,403 CassandraDaemon.java
(line 133) Exception in thread Thread[Thread-97564,5,main]
java.lang.RuntimeException: Last written key
DecoratedKey(152706250731373455824787766459206671594,
757365723a3a313a3a333434313038323239) >= current key DecoratedKey(

ERROR [Thread-661] 2013-03-15 21:02:05,981 CassandraDaemon.java (line
133) Exception in thread Thread[Thread-661,5,main]
java.lang.NegativeArraySizeException


Details:
6 node cluster
cassandra  1.2.2 - single token per node
RandomPartitioner, EC2Snitch
Replication: SimpleStrategy, rf=3
Ubuntu 10.10 x86_64
EC2 m1.large


Dane

Re: secondary index problem

[Brett Tinling]

Aaron,

No recipe yet.  It pops up randomly and, i think due to the nature of our app, 
goes away.

Seems like when we have updates that are large (10k rows in one mutate) the 
problem is more likely to occur.

I'll try to workout a repro...

-Brett

On Mar 18, 2013, at 10:18 AM, aaron morton wrote:

> Brett, 
>   Do you have some steps to reproduce the problem ? If so please create a 
> ticket on jira. 
> 
> Cheers
> 
> -
> Aaron Morton
> Freelance Cassandra Consultant
> New Zealand
> 
> @aaronmorton
> http://www.thelastpickle.com
> 
> On 16/03/2013, at 11:40 AM, Janne Jalkanen  wrote:
> 
>> 
>> This could be either of the following bugs (which might be the same thing).  
>> I get it too every time I recycle a node on 1.1.10.
>> 
>> https://issues.apache.org/jira/browse/CASSANDRA-4973
>> or
>> https://issues.apache.org/jira/browse/CASSANDRA-4785
>> 
>> /Janne
>> 
>> On Mar 15, 2013, at 23:24 , Brett Tinling  wrote:
>> 
>>> We have a CF with an indexed column 'type', but we get incomplete results 
>>> when we query that CF for all rows matching 'type'.  We can find the 
>>> missing rows if we query by key.
>>> 
>>> * we are seeing this on a small, single node, 1.2.2 instance with few rows.
>>> * we use thrift execute_cql_query, no CL is specified
>>> * none of repair, restart, compact, scrub helped
>>> 
>>> Finally, nodetool rebuild_index fixed it.  
>>> 
>>> Is index rebuild something we need to do periodically?  How often?  Is 
>>> there a way to know when it needs to be done?  Do we have to run rebuild on 
>>> all nodes?
>>> 
>>> We have not noticed this until 1.2
>>> 
>>> Regards,
>>>  - Brett
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>> 
> 



smime.p7s
Description: S/MIME cryptographic signature

Re: Waiting on read repair?

[Jasdeep Hundal]

Thanks for the info, got a couple of follow up questions, and just as
a note, this is on Cassandra 1.2.0.

I checked the flushwriter thread pool stats and saw this:
Pool NameActive   Pending  Completed   Blocked
 All time blocked
FlushWriter   1 586183
1 17582

Also, memtable_flush_queue_size is set to 4, and
memtable_flush_writers is set to 1

In my setup, I have one extremely high traffic column family that is
flushing lots of data at once (occasionally hitting hundreds of
megabytes), and several smaller cf's for which flushes involve only a
few bytes of data.

>From the following log output, it looks like the cf with the large
data load is blocking the flush of the other cf's. Would increasing
memtable_flush_queue_size (I've got plenty of memory) and
memtable_flush_writers allow the smaller cf flushes to return faster?
Or given that I see the smaller cf's being flushed when not much has
been written to them, should I try reducing
commit_log_segment_size_in_mb.

168026:2013-03-18 17:53:41,938 INFO  [OptionalTasks:1]
org.apache.cassandra.db.ColumnFamilyStore.switchMemtable
ColumnFamilyStore.java:647) - Enqueuing flush of
Memtable-data@2098591494(458518528/458518528 serialized/live bytes,
111732 ops)
168028:2013-03-18 17:53:47,064 INFO  [OptionalTasks:1]
org.apache.cassandra.db.ColumnFamilyStore.switchMemtable
(ColumnFamilyStore.java:647) - Enqueuing flush of
Memtable-metadata@252512204(2295/2295 serialized/live bytes, 64 ops)
168029:2013-03-18 17:53:47,065 INFO  [OptionalTasks:1]
org.apache.cassandra.db.ColumnFamilyStore.switchMemtable
(ColumnFamilyStore.java:647) - Enqueuing flush of
Memtable-counters@544926156(363/363 serialized/live bytes, 12 ops)
168030:2013-03-18 17:53:47,066 INFO  [OptionalTasks:1]
org.apache.cassandra.db.ColumnFamilyStore.switchMemtable
(ColumnFamilyStore.java:647) - Enqueuing flush of
Memtable-container_counters@1703633084(430/430 serialized/live bytes,
83 ops)
168032:2013-03-18 17:53:51,950 INFO  [FlushWriter:3]
org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents
(Memtable.java:458) - Completed flushing
/mnt/test/jasdeep/data/jasdeep-data-ia-720454-Data.db (391890130
bytes) for commitlog position ReplayPosition(segmentId=1363628611044,
position=21069295)
168050:2013-03-18 17:53:55,948 INFO  [FlushWriter:3]
org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents
(Memtable.java:458) - Completed flushing
/mnt/test/jasdeep/metadata/jasdeep-metadata-ia-1280-Data.db (833
bytes) for commitlog position ReplayPosition(segmentId=1363628611047,
position=4213859)
168052:2013-03-18 17:53:55,966 INFO  [FlushWriter:3]
org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents
(Memtable.java:458) - Completed flushing
/mnt/test/jasdeep/counters/jasdeep-counters-ia-1204-Data.db (342
bytes) for commitlog position ReplayPosition(segmentId=1363628611047,
position=4213859)

Thanks again,
Jasdeep



On Mon, Mar 18, 2013 at 10:24 AM, aaron morton  wrote:
> 1. With a ConsistencyLevel of quorum, does
> FBUtilities.waitForFutures() wait for read repair to complete before
> returning?
>
> No
> That's just a utility method.
> Nothing on the read path waits for Read Repair, and controlled by
> read_repair_chance CF property, it's all async to the client request.
> There is no CL, the messages are sent to individual nodes.
>
> 2. When read repair applies a mutation, it needs to obtain a lock for
> the associated memtable.
>
> What lock are you referring to?
> When Read Repair (the RowDataResolver) wants to send a mutation it uses the
> MessageServer. On the write path there is a server wide RW lock call the
> sync lock.
>
> I've seen readrepair spend a few seconds stalling in
> org.apache.cassandra.db.Table.apply).
>
> This  could be contention around the sync lock, look for blocked tasks in
> the flush writer thread pool.
>
> I did a talk on cassandra internals at Apache Con 3 weeks ago, not sure when
> the video is going to be up but here are the slides
> http://www.slideshare.net/aaronmorton/apachecon-nafeb2013
>
> Cheers
>
> -
> Aaron Morton
> Freelance Cassandra Consultant
> New Zealand
>
> @aaronmorton
> http://www.thelastpickle.com
>
> On 16/03/2013, at 12:21 PM, Jasdeep Hundal  wrote:
>
> I've got a couple of questions related issues I'm encountering using
> Cassandra under a heavy write load:
>
> 1. With a ConsistencyLevel of quorum, does
> FBUtilities.waitForFutures() wait for read repair to complete before
> returning?
> 2. When read repair applies a mutation, it needs to obtain a lock for
> the associated memtable. Does compaction obtain this same lock? (I'm
> asking because I've seen readrepair spend a few seconds stalling in
> org.apache.cassandra.db.Table.apply).
>
> Thanks,
> Jasdeep
>
>

Re: Cassandra Compression and Wide Rows

[Drew Kutcharian]

That's what I originally thought but the OOYALA presentation from C*2012 got me 
confused. Do you guys know what's going on here?

The video: 
http://www.youtube.com/watch?v=r2nGBUuvVmc&feature=player_detailpage#t=790s
The slides: Slide 22 @ 
http://www.datastax.com/wp-content/uploads/2012/08/C2012-Hastur-NoahGibbs.pdf

-- Drew


On Mar 18, 2013, at 6:14 AM, Edward Capriolo  wrote:

> 
> Imho it is probably more efficient for wide. When you decompress 8k blocks to 
> get at a 200 byte row you create overhead , particularly young gen.
> On Monday, March 18, 2013, Sylvain Lebresne  wrote:
> > The way compression is implemented, it is oblivious to the CF being 
> > wide-row or narrow-row. There is nothing intrinsically less efficient in 
> > the compression for wide-rows.
> > --
> > Sylvain
> >
> > On Fri, Mar 15, 2013 at 11:53 PM, Drew Kutcharian  wrote:
> >>
> >> Hey Guys,
> >>
> >> I remember reading somewhere that C* compression is not very effective 
> >> when most of the CFs are in wide-row format and some folks turn the 
> >> compression off and use disk level compression as a workaround. 
> >> Considering that wide rows with composites are "first class citizens" in 
> >> CQL3, is this still the case? Has there been any improvements on this?
> >>
> >> Thanks,
> >>
> >> Drew
> >

Re: Cassandra Compression and Wide Rows

[Edward Capriolo]

I feel this has come up before. I believe the compression is block based,
so just because no two column names are the same does not mean the
compression will not be effective. Possibly in their case the compression
was not effective.

On Mon, Mar 18, 2013 at 9:08 PM, Drew Kutcharian  wrote:

> That's what I originally thought but the OOYALA presentation from C*2012
> got me confused. Do you guys know what's going on here?
>
> The video:
> http://www.youtube.com/watch?v=r2nGBUuvVmc&feature=player_detailpage#t=790s
> The slides: Slide 22 @
> http://www.datastax.com/wp-content/uploads/2012/08/C2012-Hastur-NoahGibbs.pdf
>
> -- Drew
>
>
> On Mar 18, 2013, at 6:14 AM, Edward Capriolo 
> wrote:
>
>
> Imho it is probably more efficient for wide. When you decompress 8k blocks
> to get at a 200 byte row you create overhead , particularly young gen.
> On Monday, March 18, 2013, Sylvain Lebresne  wrote:
> > The way compression is implemented, it is oblivious to the CF being
> wide-row or narrow-row. There is nothing intrinsically less efficient in
> the compression for wide-rows.
> > --
> > Sylvain
> >
> > On Fri, Mar 15, 2013 at 11:53 PM, Drew Kutcharian 
> wrote:
> >>
> >> Hey Guys,
> >>
> >> I remember reading somewhere that C* compression is not very effective
> when most of the CFs are in wide-row format and some folks turn the
> compression off and use disk level compression as a workaround. Considering
> that wide rows with composites are "first class citizens" in CQL3, is this
> still the case? Has there been any improvements on this?
> >>
> >> Thanks,
> >>
> >> Drew
> >
>
>
>

Re: Cassandra Compression and Wide Rows

[Drew Kutcharian]

Edward/Sylvain,

I also came across this post on DataStax's blog:

> When to use compression
> Compression is best suited for ColumnFamilies where there are many rows, with 
> each row having the same columns, or at least many columns in common. For 
> example, a ColumnFamily containing user data such as username, email, etc., 
> would be a good candidate for compression. The more similar the data across 
> rows, the greater the compression ratio will be, and the larger the gain in 
> read performance.
> Compression is not as good a fit for ColumnFamilies where each row has a 
> different set of columns, or where there are just a few very wide rows. 
> Dynamic column families such as this will not yield good compression ratios.

http://www.datastax.com/dev/blog/whats-new-in-cassandra-1-0-compression

@Sylvain, does this still apply on more recent versions of C*?


-- Drew



On Mar 18, 2013, at 7:16 PM, Edward Capriolo  wrote:

> I feel this has come up before. I believe the compression is block based, so 
> just because no two column names are the same does not mean the compression 
> will not be effective. Possibly in their case the compression was not 
> effective.
> 
> On Mon, Mar 18, 2013 at 9:08 PM, Drew Kutcharian  wrote:
> That's what I originally thought but the OOYALA presentation from C*2012 got 
> me confused. Do you guys know what's going on here?
> 
> The video: 
> http://www.youtube.com/watch?v=r2nGBUuvVmc&feature=player_detailpage#t=790s
> The slides: Slide 22 @ 
> http://www.datastax.com/wp-content/uploads/2012/08/C2012-Hastur-NoahGibbs.pdf
> 
> -- Drew
> 
> 
> On Mar 18, 2013, at 6:14 AM, Edward Capriolo  wrote:
> 
>> 
>> Imho it is probably more efficient for wide. When you decompress 8k blocks 
>> to get at a 200 byte row you create overhead , particularly young gen.
>> On Monday, March 18, 2013, Sylvain Lebresne  wrote:
>> > The way compression is implemented, it is oblivious to the CF being 
>> > wide-row or narrow-row. There is nothing intrinsically less efficient in 
>> > the compression for wide-rows.
>> > --
>> > Sylvain
>> >
>> > On Fri, Mar 15, 2013 at 11:53 PM, Drew Kutcharian  wrote:
>> >>
>> >> Hey Guys,
>> >>
>> >> I remember reading somewhere that C* compression is not very effective 
>> >> when most of the CFs are in wide-row format and some folks turn the 
>> >> compression off and use disk level compression as a workaround. 
>> >> Considering that wide rows with composites are "first class citizens" in 
>> >> CQL3, is this still the case? Has there been any improvements on this?
>> >>
>> >> Thanks,
>> >>
>> >> Drew
>> >
> 
> 

Cassandra 1.2.0H startup exception

[杨辉强]

Hi, all:

   I am a beginner of cassandra. I have a four node cassandra group.
   One of my cassandra group node had been running for a week. Recently 
because of too much writing and reading, it crashed.

I want to restart the node.
   While I start up cassandra, it ends up with throwing exception as 
follow every time:

java.lang.RuntimeException: The provided key was not UTF8 encoded.
at 
org.apache.cassandra.dht.OrderPreservingPartitioner.getToken(OrderPreservingPartitioner.java:170)
at 
org.apache.cassandra.dht.OrderPreservingPartitioner.decorateKey(OrderPreservingPartitioner.java:42)

at org.apache.cassandra.db.Table.apply(Table.java:380)
at org.apache.cassandra.db.Table.apply(Table.java:354)
at 
org.apache.cassandra.db.commitlog.CommitLogReplayer$1.runMayThrow(CommitLogReplayer.java:260)
at 
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)

at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

at java.lang.Thread.run(Thread.java:662)

  I have googled and read the archive. But I can't find answer, please 
help me. Thank you very much.

Re: Cassandra 1.2.0H startup exception

[杨辉强]

On 03/19/2013 11:22 AM, 杨辉强 wrote:

Hi, all:

   I am a beginner of cassandra. I have a four node cassandra group.
   One of my cassandra group node had been running for a week. 
Recently because of too much writing and reading, it crashed.

I want to restart the node.
   While I start up cassandra, it ends up with throwing exception as 
follow every time:

java.lang.RuntimeException: The provided key was not UTF8 encoded.
at 
org.apache.cassandra.dht.OrderPreservingPartitioner.getToken(OrderPreservingPartitioner.java:170)
at 
org.apache.cassandra.dht.OrderPreservingPartitioner.decorateKey(OrderPreservingPartitioner.java:42)

at org.apache.cassandra.db.Table.apply(Table.java:380)
at org.apache.cassandra.db.Table.apply(Table.java:354)
at 
org.apache.cassandra.db.commitlog.CommitLogReplayer$1.runMayThrow(CommitLogReplayer.java:260)
at 
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)

at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

at java.lang.Thread.run(Thread.java:662)

  I have googled and read the archive. But I can't find answer, please 
help me. Thank you very much.



More crash log:

ERROR [main] 2013-03-18 18:02:47,095 CassandraDaemon.java (line 414) 
Exception encountered during startup
java.lang.RuntimeException: java.util.concurrent.ExecutionException: 
java.lang.RuntimeException: The provided key was not UTF8 encoded.
at 
org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:377)
at 
org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:366)
at 
org.apache.cassandra.db.commitlog.CommitLogReplayer.blockForWrites(CommitLogReplayer.java:106)
at 
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:147)
at 
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:126)
at 
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:268)
at 
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:397)
at 
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:440)
Caused by: java.util.concurrent.ExecutionException: 
java.lang.RuntimeException: The provided key was not UTF8 encoded.
at 
java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)

at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at 
org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:373)

... 7 more
Caused by: java.lang.RuntimeException: The provided key was not UTF8 
encoded.
at 
org.apache.cassandra.dht.OrderPreservingPartitioner.getToken(OrderPreservingPartitioner.java:170)
at 
org.apache.cassandra.dht.OrderPreservingPartitioner.decorateKey(OrderPreservingPartitioner.java:42)

at org.apache.cassandra.db.Table.apply(Table.java:380)
at org.apache.cassandra.db.Table.apply(Table.java:354)
at 
org.apache.cassandra.db.commitlog.CommitLogReplayer$1.runMayThrow(CommitLogReplayer.java:260)
at 
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at 
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)

at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

at java.lang.Thread.run(Thread.java:662)
Caused by: java.nio.charset.MalformedInputException: Input length = 1
at 
java.nio.charset.CoderResult.throwException(CoderResult.java:260)

at java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:781)
at 
org.apache.cassandra.utils.ByteBufferUtil.string(ByteBufferUtil.java:167)
at 
org.apache.cassandra.utils.ByteBufferUtil.string(ByteBufferUtil.java:124)
at 
org.apache.cassandra.dht.OrderPreservingPartitioner.getToken(OrderPreservingPartitioner.java:166)

... 11 more