Re: Cassandra 1.20 with Cloudera Hadoop (CDH4) Compatibility Issue

[Yang Song]

Thanks Michael. I attached the reply I got back from CDH4 user group from
Harsh. Hope to share the experience.
"
In CDH4, the MR1 and MR2 APIs are both fully compatible (such that
moving to YARN in future would require no recompilation from MR1
produced jars). You can consider it "2.0" API in binary form, and not
0.20 exactly (i.e. its not backwards compatible with CDH3).

Cassandra is distributing binaries built on MR1 (Apache Hadoop 1,
CDH3, etc.), which wouldn't work on your CDH4 platform. You will have
to recompile against the proper platform to get binary-compatible
jars/etc.."

Interesting. Has anyone have issue with CDH4 with the newly released C*
1.21?

Thanks

2013/2/15 Michael Kjellman 

> Sorry. I meant to say even though there *wasnt* a major change between
> 1.0.x and 0.22. The big change was 0.20 to 0.22. Sorry for the confusion.
>
> On Feb 15, 2013, at 9:53 PM, "Michael Kjellman" 
> wrote:
>
> There were pretty big changes in Hadoop between 0.20 and 0.22 (which is
> now known as 1.0.x) even though there were major change between 0.22 and
> 1.0.x. Cloudera hadn't yet upgraded to 0.22 which uses the new map reduce
> framework instead of the old mapred API. I don't see the C* project back
> porting their code at this time and if anything Cloudera should update
> their release!!
>
> On Feb 15, 2013, at 9:48 PM, "Yang Song"  wrote:
>
> It is interesting though. I am using CDH4 which contains hadoop 0.20, and
> I am using Cassandra 1.20.
> The previous mentioned errors still occur. Any suggestions? Thanks.
>
> 2013/2/15 Michael Kjellman 
>
>> That bug is kinda wrong though. 1.0.x is current for like a year now and
>> C* works great with it :)
>>
>> On Feb 15, 2013, at 7:38 PM, "Dave Brosius" 
>> wrote:
>>
>>  see https://issues.apache.org/jira/browse/CASSANDRA-5201
>>
>>
>> On 02/15/2013 10:05 PM, Yang Song wrote:
>>
>>   Hi,
>>
>>  Does anyone use CDH4's Hadoop with Cassandra to interact? The goal is
>> simply read/write to Cassandra from Hadoop direclty using
>> ColumnFamilyInput(Output)Format, but seems a bit compatibility issue. There
>> are two java exceptions
>>
>>  1. java.lang.IncompatibleClassChangeError: Found interface
>> org.apache.hadoop.mapreduce.JobContext, but class was expected
>> This shows when I run hadoop jar file to read directly from Cassandra.
>> Seems that there is a change on Hadoop that JobContext was changed from
>> class to interface. Has anyone have similar issue?
>> Does it mean the Hadoop version in CDH4 is old?
>>
>>  2. Another error is java.lang.NoSuchMethodError:
>> org.apache.cassandra.hadoop.ConfigHelper.setRpcPort(Lorg/apache/hadoop/conf/Configuration;Ljava/lang/String;)V
>> This shows when the jar file contains rpc port for remote Cassandra
>> cluster.
>>
>>  Does anyone have similiar experience? Any comments are welcome. thanks!
>>
>>
>>
>

Re: Cassandra 1.20 with Cloudera Hadoop (CDH4) Compatibility Issue

[Edward Capriolo]

Here is the deal.

http://wiki.apache.org/hadoop/Defining%20Hadoop

INAPPROPRIATE: Automotive Joe's Crankshaft: 100% compatible with Hadoop

Bad, because "100% compatible" is a meaningless statement. Even Apache
releases have regressions; cases were versions are incompatible *even
when the Java interfaces don't change*. A statement about
compatibility ought to be qualilified "Certified by Joe's brother Bob
as 100% compatible with Apache Hadoop(TM)". In the US, the marketing
team may be able to get way with the "100% compatible" claim, but in
some EU countries, sticking that statement up your web site is a claim
that residents can demand the vendor justifies, or take it down.

So as a result, if you are running something NOT apache hadoop, CDH,
DSE, or whatever they are NOT compatible with hadoop or each other by
definition.

Anyway, I have been using hadoop for years, and its biggest problem is
that it has never become happy with its own codebase. Old api, new
api, jobtracker, yarn, all these thing change, there is really no
upgrade/downgrade path because there are so many branches etc.Open
source products move swiftly and end users are normally left holding
the ball in figuring it our how to do it sanely. With Cassandra +
Hadoop it is "double trouble".

All that being said I think it is unrealistic to count on vendors 100%
to solve your problems. If something throws you and exception like..

org.apache.cassandra.hadoop.ConfigHelper.setRpcPort(Lorg/apache/hadoop/conf/Configuration;Ljava/lang/String;)V

Guess what? It is time to get out your compiler.

On Sat, Feb 16, 2013 at 3:39 AM, Yang Song  wrote:
> Thanks Michael. I attached the reply I got back from CDH4 user group from
> Harsh. Hope to share the experience.
> "
> In CDH4, the MR1 and MR2 APIs are both fully compatible (such that
> moving to YARN in future would require no recompilation from MR1
> produced jars). You can consider it "2.0" API in binary form, and not
> 0.20 exactly (i.e. its not backwards compatible with CDH3).
>
> Cassandra is distributing binaries built on MR1 (Apache Hadoop 1,
> CDH3, etc.), which wouldn't work on your CDH4 platform. You will have
> to recompile against the proper platform to get binary-compatible
> jars/etc.."
>
> Interesting. Has anyone have issue with CDH4 with the newly released C*
> 1.21?
>
> Thanks
>
> 2013/2/15 Michael Kjellman 
>>
>> Sorry. I meant to say even though there *wasnt* a major change between
>> 1.0.x and 0.22. The big change was 0.20 to 0.22. Sorry for the confusion.
>>
>> On Feb 15, 2013, at 9:53 PM, "Michael Kjellman" 
>> wrote:
>>
>> There were pretty big changes in Hadoop between 0.20 and 0.22 (which is
>> now known as 1.0.x) even though there were major change between 0.22 and
>> 1.0.x. Cloudera hadn't yet upgraded to 0.22 which uses the new map reduce
>> framework instead of the old mapred API. I don't see the C* project back
>> porting their code at this time and if anything Cloudera should update their
>> release!!
>>
>> On Feb 15, 2013, at 9:48 PM, "Yang Song"  wrote:
>>
>> It is interesting though. I am using CDH4 which contains hadoop 0.20, and
>> I am using Cassandra 1.20.
>> The previous mentioned errors still occur. Any suggestions? Thanks.
>>
>> 2013/2/15 Michael Kjellman 
>>>
>>> That bug is kinda wrong though. 1.0.x is current for like a year now and
>>> C* works great with it :)
>>>
>>> On Feb 15, 2013, at 7:38 PM, "Dave Brosius" 
>>> wrote:
>>>
>>> see https://issues.apache.org/jira/browse/CASSANDRA-5201
>>>
>>>
>>> On 02/15/2013 10:05 PM, Yang Song wrote:
>>>
>>> Hi,
>>>
>>> Does anyone use CDH4's Hadoop with Cassandra to interact? The goal is
>>> simply read/write to Cassandra from Hadoop direclty using
>>> ColumnFamilyInput(Output)Format, but seems a bit compatibility issue. There
>>> are two java exceptions
>>>
>>> 1. java.lang.IncompatibleClassChangeError: Found interface
>>> org.apache.hadoop.mapreduce.JobContext, but class was expected
>>> This shows when I run hadoop jar file to read directly from Cassandra.
>>> Seems that there is a change on Hadoop that JobContext was changed from
>>> class to interface. Has anyone have similar issue?
>>> Does it mean the Hadoop version in CDH4 is old?
>>>
>>> 2. Another error is java.lang.NoSuchMethodError:
>>> org.apache.cassandra.hadoop.ConfigHelper.setRpcPort(Lorg/apache/hadoop/conf/Configuration;Ljava/lang/String;)V
>>> This shows when the jar file contains rpc port for remote Cassandra
>>> cluster.
>>>
>>> Does anyone have similiar experience? Any comments are welcome. thanks!
>>>
>>>
>>
>

Re: virtual nodes + map reduce = too many mappers

[Edward Capriolo]

No one had ever tried vnodes with hadoop until the OP did, or they
would have noticed this. No one extensively used it with secondary
indexes either from the last ticket I mentioned.

My mistake they are not a default.

I do think vnodes are awesome, its great that c* has the longer
release cylcle. Just saying I do not know what .0 and .1 releases are.
They just seem like extended beta-s to me.

Edward


On Fri, Feb 15, 2013 at 11:10 PM, Eric Evans  wrote:
> On Fri, Feb 15, 2013 at 7:01 PM, Edward Capriolo  
> wrote:
>> Seems like the hadoop Input format should combine the splits that are
>> on the same node into the same map task, like Hadoop's
>> CombinedInputFormat can. I am not sure who recommends vnodes as the
>> default, because this is now the second problem (that I know of) of
>> this class where vnodes has extra overhead,
>> https://issues.apache.org/jira/browse/CASSANDRA-5161
>>
>> This seems to be the standard operating practice in c* now, enable
>> things in the default configuration like new partitioners and newer
>> features like vnodes, even though they are not heavily tested in the
>> wild or well understood, then deal with fallout.
>
> Except that it is not in fact enabled by default; The default remains
> 1-token-per-node.
>
> That said, the only way that a feature like this will ever be heavily
> tested in the wild, and well understood, is if it is actually put to
> use.  Speaking only for myself, I am grateful to users like Cem who
> test new features and report the issues they find.
>
>> On Fri, Feb 15, 2013 at 11:52 AM, cem  wrote:
>>> Hi All,
>>>
>>> I have just started to use virtual nodes. I set the number of nodes to 256
>>> as recommended.
>>>
>>> The problem that I have is when I run a mapreduce job it creates node * 256
>>> mappers. It creates node * 256 splits. this effects the performance since
>>> the range queries have a lot of overhead.
>>>
>>> Any suggestion to improve the performance? It seems like I need to lower the
>>> number of virtual nodes.
>>>
>>> Best Regards,
>>> Cem
>>>
>>>
>
>
>
> --
> Eric Evans
> Acunu | http://www.acunu.com | @acunu

Re: virtual nodes + map reduce = too many mappers

[Eric Evans]

On Sat, Feb 16, 2013 at 9:13 AM, Edward Capriolo  wrote:
> No one had ever tried vnodes with hadoop until the OP did, or they
> would have noticed this. No one extensively used it with secondary
> indexes either from the last ticket I mentioned.
>
> My mistake they are not a default.
>
> I do think vnodes are awesome, its great that c* has the longer
> release cylcle. Just saying I do not know what .0 and .1 releases are.
> They just seem like extended beta-s to me.

We should definitely aspire to better/more thorough QA, but at the
risk of making what sounds like an excuse, I would argue that this is
the nature of open source software development.  You "Release Early,
Release Often", and iterate with your early adopters to shake out the
missed bugs.

What's important, I think, is to minimize the impact on existing
users, and properly set expectations.  I don't see where we've failed
here, but I'm definitely open to hearing that I'm wrong (or how we
could have done better).

> On Fri, Feb 15, 2013 at 11:10 PM, Eric Evans  wrote:
>> On Fri, Feb 15, 2013 at 7:01 PM, Edward Capriolo  
>> wrote:
>>> Seems like the hadoop Input format should combine the splits that are
>>> on the same node into the same map task, like Hadoop's
>>> CombinedInputFormat can. I am not sure who recommends vnodes as the
>>> default, because this is now the second problem (that I know of) of
>>> this class where vnodes has extra overhead,
>>> https://issues.apache.org/jira/browse/CASSANDRA-5161
>>>
>>> This seems to be the standard operating practice in c* now, enable
>>> things in the default configuration like new partitioners and newer
>>> features like vnodes, even though they are not heavily tested in the
>>> wild or well understood, then deal with fallout.
>>
>> Except that it is not in fact enabled by default; The default remains
>> 1-token-per-node.
>>
>> That said, the only way that a feature like this will ever be heavily
>> tested in the wild, and well understood, is if it is actually put to
>> use.  Speaking only for myself, I am grateful to users like Cem who
>> test new features and report the issues they find.
>>
>>> On Fri, Feb 15, 2013 at 11:52 AM, cem  wrote:
 Hi All,

 I have just started to use virtual nodes. I set the number of nodes to 256
 as recommended.

 The problem that I have is when I run a mapreduce job it creates node * 256
 mappers. It creates node * 256 splits. this effects the performance since
 the range queries have a lot of overhead.

 Any suggestion to improve the performance? It seems like I need to lower 
 the
 number of virtual nodes.

 Best Regards,
 Cem


>>
>>
>>
>> --
>> Eric Evans
>> Acunu | http://www.acunu.com | @acunu



-- 
Eric Evans
Acunu | http://www.acunu.com | @acunu

Re: Cassandra 1.20 with Cloudera Hadoop (CDH4) Compatibility Issue

[Jeremy Hanna]

Fwiw - here is are some changes that a friend said should make C*'s Hadoop 
support work with CDH4 - for ColumnFamilyRecordReader.
https://gist.github.com/jeromatron/4967799

On Feb 16, 2013, at 8:23 AM, Edward Capriolo  wrote:

> Here is the deal.
> 
> http://wiki.apache.org/hadoop/Defining%20Hadoop
> 
> INAPPROPRIATE: Automotive Joe's Crankshaft: 100% compatible with Hadoop
> 
> Bad, because "100% compatible" is a meaningless statement. Even Apache
> releases have regressions; cases were versions are incompatible *even
> when the Java interfaces don't change*. A statement about
> compatibility ought to be qualilified "Certified by Joe's brother Bob
> as 100% compatible with Apache Hadoop(TM)". In the US, the marketing
> team may be able to get way with the "100% compatible" claim, but in
> some EU countries, sticking that statement up your web site is a claim
> that residents can demand the vendor justifies, or take it down.
> 
> So as a result, if you are running something NOT apache hadoop, CDH,
> DSE, or whatever they are NOT compatible with hadoop or each other by
> definition.
> 
> Anyway, I have been using hadoop for years, and its biggest problem is
> that it has never become happy with its own codebase. Old api, new
> api, jobtracker, yarn, all these thing change, there is really no
> upgrade/downgrade path because there are so many branches etc.Open
> source products move swiftly and end users are normally left holding
> the ball in figuring it our how to do it sanely. With Cassandra +
> Hadoop it is "double trouble".
> 
> All that being said I think it is unrealistic to count on vendors 100%
> to solve your problems. If something throws you and exception like..
> 
> org.apache.cassandra.hadoop.ConfigHelper.setRpcPort(Lorg/apache/hadoop/conf/Configuration;Ljava/lang/String;)V
> 
> Guess what? It is time to get out your compiler.
> 
> On Sat, Feb 16, 2013 at 3:39 AM, Yang Song  wrote:
>> Thanks Michael. I attached the reply I got back from CDH4 user group from
>> Harsh. Hope to share the experience.
>> "
>> In CDH4, the MR1 and MR2 APIs are both fully compatible (such that
>> moving to YARN in future would require no recompilation from MR1
>> produced jars). You can consider it "2.0" API in binary form, and not
>> 0.20 exactly (i.e. its not backwards compatible with CDH3).
>> 
>> Cassandra is distributing binaries built on MR1 (Apache Hadoop 1,
>> CDH3, etc.), which wouldn't work on your CDH4 platform. You will have
>> to recompile against the proper platform to get binary-compatible
>> jars/etc.."
>> 
>> Interesting. Has anyone have issue with CDH4 with the newly released C*
>> 1.21?
>> 
>> Thanks
>> 
>> 2013/2/15 Michael Kjellman 
>>> 
>>> Sorry. I meant to say even though there *wasnt* a major change between
>>> 1.0.x and 0.22. The big change was 0.20 to 0.22. Sorry for the confusion.
>>> 
>>> On Feb 15, 2013, at 9:53 PM, "Michael Kjellman" 
>>> wrote:
>>> 
>>> There were pretty big changes in Hadoop between 0.20 and 0.22 (which is
>>> now known as 1.0.x) even though there were major change between 0.22 and
>>> 1.0.x. Cloudera hadn't yet upgraded to 0.22 which uses the new map reduce
>>> framework instead of the old mapred API. I don't see the C* project back
>>> porting their code at this time and if anything Cloudera should update their
>>> release!!
>>> 
>>> On Feb 15, 2013, at 9:48 PM, "Yang Song"  wrote:
>>> 
>>> It is interesting though. I am using CDH4 which contains hadoop 0.20, and
>>> I am using Cassandra 1.20.
>>> The previous mentioned errors still occur. Any suggestions? Thanks.
>>> 
>>> 2013/2/15 Michael Kjellman 
 
 That bug is kinda wrong though. 1.0.x is current for like a year now and
 C* works great with it :)
 
 On Feb 15, 2013, at 7:38 PM, "Dave Brosius" 
 wrote:
 
 see https://issues.apache.org/jira/browse/CASSANDRA-5201
 
 
 On 02/15/2013 10:05 PM, Yang Song wrote:
 
 Hi,
 
 Does anyone use CDH4's Hadoop with Cassandra to interact? The goal is
 simply read/write to Cassandra from Hadoop direclty using
 ColumnFamilyInput(Output)Format, but seems a bit compatibility issue. There
 are two java exceptions
 
 1. java.lang.IncompatibleClassChangeError: Found interface
 org.apache.hadoop.mapreduce.JobContext, but class was expected
 This shows when I run hadoop jar file to read directly from Cassandra.
 Seems that there is a change on Hadoop that JobContext was changed from
 class to interface. Has anyone have similar issue?
 Does it mean the Hadoop version in CDH4 is old?
 
 2. Another error is java.lang.NoSuchMethodError:
 org.apache.cassandra.hadoop.ConfigHelper.setRpcPort(Lorg/apache/hadoop/conf/Configuration;Ljava/lang/String;)V
 This shows when the jar file contains rpc port for remote Cassandra
 cluster.
 
 Does anyone have similiar experience? Any comments are welcome. thanks!
 
 
>>> 
>> 

Re: Size Tiered -> Leveled Compaction

[Mike]

Another piece of information that would be useful is advice on how to 
properly set the SSTable size for your usecase.  I understand the 
default is 5MB, a lot of examples show the use of 10MB, and I've seen 
cases where people have set is as high as 200MB.


Any information is appreciated,
-Mike

On 2/14/2013 4:10 PM, Michael Theroux wrote:
BTW, when I say "major compaction", I mean running the "nodetool 
compact" command (which does a major compaction for Sized Tiered 
Compaction).  I didn't see the distribution of SSTables I expected 
until I ran that command, in the steps I described below.


-Mike

On Feb 14, 2013, at 3:51 PM, Wei Zhu wrote:


I haven't tried to switch compaction strategy. We started with LCS.

For us, after massive data imports (5000 w/seconds for 6 days), the 
first repair is painful since there is quite some data inconsistency. 
For 150G nodes, repair brought in about 30 G and created thousands of 
pending compactions. It took almost a day to clear those. Just be 
prepared LCS is really slow in 1.1.X. System performance degrades 
during that time since reads could go to more SSTable, we see 20 
SSTable lookup for one read.. (We tried everything we can and 
couldn't speed it up. I think it's single threaded and it's not 
recommended to turn on multithread compaction. We even tried that, it 
didn't help )There is parallel LCS in 1.2 which is supposed to 
alleviate the pain. Haven't upgraded yet, hope it works:)


http://www.datastax.com/dev/blog/performance-improvements-in-cassandra-1-2


Since our cluster is not write intensive, only 100 w/seconds. I don't 
see any pending compactions during regular operation.


One thing worth mentioning is the size of the SSTable, default is 5M 
which is kind of small for 200G (all in one CF) data set, and we are 
on SSD.  It more than  150K files in one directory. (200G/5M = 40K 
SSTable and each SSTable creates 4 files on disk)  You might want to 
watch that and decide the SSTable size.


By the way, there is no concept of Major compaction for LCS. Just for 
fun, you can look at a file called $CFName.json in your data 
directory and it tells you the SSTable distribution among different 
levels.


-Wei


*From:* Charles Brophy mailto:cbro...@zulily.com>>
*To:* user@cassandra.apache.org 
*Sent:* Thursday, February 14, 2013 8:29 AM
*Subject:* Re: Size Tiered -> Leveled Compaction

I second these questions: we've been looking into changing some of 
our CFs to use leveled compaction as well. If anybody here has the 
wisdom to answer them it would be of wonderful help.


Thanks
Charles

On Wed, Feb 13, 2013 at 7:50 AM, Mike > wrote:


Hello,

I'm investigating the transition of some of our column families
from Size Tiered -> Leveled Compaction.  I believe we have some
high-read-load column families that would benefit tremendously.

I've stood up a test DB Node to investigate the transition.  I
successfully alter the column family, and I immediately noticed a
large number (1000+) pending compaction tasks become available,
but no compaction get executed.

I tried running "nodetool sstableupgrade" on the column family,
and the compaction tasks don't move.

I also notice no changes to the size and distribution of the
existing SSTables.

I then run a major compaction on the column family.  All pending
compaction tasks get run, and the SSTables have a distribution
that I would expect from LeveledCompaction (lots and lots of 10MB
files).

Couple of questions:

1) Is a major compaction required to transition from size-tiered
to leveled compaction?
2) Are major compactions as much of a concern for
LeveledCompaction as their are for Size Tiered?

All the documentation I found concerning transitioning from Size
Tiered to Level compaction discuss the alter table cql command,
but I haven't found too much on what else needs to be done after
the schema change.

I did these tests with Cassandra 1.1.9.

Thanks,
-Mike

Re: RuntimeException during leveled compaction

[aaron morton]

That sounds like something wrong with the way the rows are merged during 
compaction then. 

Can you run the compaction with DEBUG logging and raise a ticket? You may want 
to do this with the node not in the ring. Five minutes after it starts it will 
run pending compactions, so if there if compactions are not running they should 
start again. 

Cheers
 
-
Aaron Morton
Freelance Cassandra Developer
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 13/02/2013, at 8:11 PM, Andre Sprenger  wrote:

> 
> Aaron,
> 
> thanks for your help. 
> 
> I ran 'nodetool scrub' and it finished after a couple of hours. But there are 
> no infos about 
> out of order rows in the logs and the compaction on the column family still 
> raises the same
> exception. 
> 
> With the row key I could identify some of the errant SSTables and removed 
> them during
> a node restart. On some nodes compaction is working for the moment but there 
> are likely
> more corrupt datafiles and than I would be in the same situation as before.
> 
> So I still need some help to resolve this issue!
> 
> Cheers
> Andre
> 
> 
> 2013/2/12 aaron morton 
> snapshot all nodes so you have a backup: nodetool snapshot -t corrupt
> 
> run nodetool scrub on the errant CF. 
> 
> Look for messages such as:
> 
> "Out of order row detected…"
> "%d out of order rows found while scrubbing %s; Those have been written (in 
> order) to a new sstable (%s)"
> 
> In the logs. 
> 
> Cheers
>   
> -
> Aaron Morton
> Freelance Cassandra Developer
> New Zealand
> 
> @aaronmorton
> http://www.thelastpickle.com
> 
> On 12/02/2013, at 6:13 AM, Andre Sprenger  wrote:
> 
>> Hi,
>> 
>> I'm running a 6 node Cassandra 1.1.5 cluster on EC2. We have switched to 
>> leveled compaction a couple of weeks ago, 
>> this has been successful. Some days ago 3 of the nodes start to log the 
>> following exception during compaction of 
>> a particular column family:
>> 
>> ERROR [CompactionExecutor:726] 2013-02-11 13:02:26,582 
>> AbstractCassandraDaemon.java (line 135) Exception in thread 
>> Thread[CompactionExecutor:726,1,main]
>> java.lang.RuntimeException: Last written key 
>> DecoratedKey(84590743047470232854915142878708713938, 
>> 3133353533383530323237303130313030303232313537303030303132393832) 
>> >= current key DecoratedKey(28357704665244162161305918843747894551, 
>> >31333430313336313830333831303130313030303230313632303030303036363338) 
>> writing into 
>> /var/cassandra/data/AdServer/EventHistory/Adserver-EventHistory-tmp-he-68638-Data.db
>> at 
>> org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:134)
>> at 
>> org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:153)
>> at 
>> org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:159)
>> at 
>> org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:50)
>> at 
>> org.apache.cassandra.db.compaction.CompactionManager$1.runMayThrow(CompactionManager.java:154)
>> at 
>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>> at 
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>> 
>> Compaction does not happen any more for the column family and read 
>> performance gets worse because of the growing 
>> number of data files accessed during reads. Looks like one or more of the 
>> data files are corrupt and have keys
>> that are stored out of order.
>> 
>> Any help to resolve this situation would be greatly appreciated.
>> 
>> Thanks
>> Andre
>> 
> 
> 

Re: Deleting old items

[aaron morton]

>  Is that a feature that could possibly be developed one day ?
No. 
Timestamps are essentially internal implementation used to resolve different 
values for the same column. 

> With "min_compaction_level_threshold" did you mean "min_compaction_threshold" 
>  ? If so, why should I do that, what are the advantage/inconvenient of 
> reducing this value ?
Yes, min_compaction_threshold, my bad. 
If you have a wide row and delete a lot of values you will end up with a lot of 
tombstones. These may dramatically reduce the read performance until they are 
purged. Reducing the compaction threshold makes compaction happen more 
frequently. 

> Looking at the doc I saw that: "max_compaction_threshold: Ignored in 
> Cassandra 1.1 and later.". How to ensure that I'll always keep a small amount 
> of SSTables then ?
AFAIK it's not. 
There may be some confusion about the location of the settings in CLI vs CQL. 
Can you point to the docs. 

Cheers

-
Aaron Morton
Freelance Cassandra Developer
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 13/02/2013, at 10:14 PM, Alain RODRIGUEZ  wrote:

> Hi Aaron, once again thanks for this answer.
>> "So is it possible to delete all the data inserted in some CF between 2 
>> dates or data older than 1 month ?"
> "No. "
> 
> Why is there no way of deleting or getting data using the internal timestamp 
> stored alongside of any inserted column (as described here: 
> http://www.datastax.com/docs/1.1/ddl/column_family#standard-columns) ? Is 
> that a feature that could possibly be developed one day ? It could be useful 
> to perform delete of old data or to bring to a dev cluster just the last week 
> of data for example.
> 
> With "min_compaction_level_threshold" did you mean "min_compaction_threshold" 
>  ? If so, why should I do that, what are the advantage/inconvenient of 
> reducing this value ?
> 
> Looking at the doc I saw that: "max_compaction_threshold: Ignored in 
> Cassandra 1.1 and later.". How to ensure that I'll always keep a small amount 
> of SSTables then ? Why is this deprecated ?
> 
> Alain
> 
> 
> 2013/2/12 aaron morton 
>> So is it possible to delete all the data inserted in some CF between 2 dates 
>> or data older than 1 month ?
> No. 
> 
> You need to issue row level deletes. If you don't know the row key you'll 
> need to do range scans to locate them. 
> 
> If you are deleting parts of wide rows consider reducing the 
> min_compaction_level_threshold on the CF to 2
> 
> Cheers
> 
> 
> -
> Aaron Morton
> Freelance Cassandra Developer
> New Zealand
> 
> @aaronmorton
> http://www.thelastpickle.com
> 
> On 12/02/2013, at 4:21 AM, Alain RODRIGUEZ  wrote:
> 
>> Hi,
>> 
>> I would like to know if there is a way to delete old/unused data easily ?
>> 
>> I know about TTL but there are 2 limitations of TTL:
>> 
>> - AFAIK, there is no TTL on counter columns
>> - TTL need to be defined at write time, so it's too late for data already 
>> inserted.
>> 
>> I also could use a standard "delete" but it seems inappropriate for such a 
>> massive.
>> 
>> In some cases, I don't know the row key and would like to delete all the 
>> rows starting by, let's say, "1050#..." 
>> 
>> Even better, I understood that columns are always inserted in C* with (name, 
>> value, timestamp). So is it possible to delete all the data inserted in some 
>> CF between 2 dates or data older than 1 month ?
>> 
>> Alain
> 
> 

Re: Deleting old items

[Alain RODRIGUEZ]

"Can you point to the docs."

http://www.datastax.com/docs/1.1/configuration/storage_configuration#max-compaction-threshold

And thanks about the rest of your answers, once again ;-).

Alain


2013/2/16 aaron morton 

>  Is that a feature that could possibly be developed one day ?
>
> No.
> Timestamps are essentially internal implementation used to resolve
> different values for the same column.
>
> With "min_compaction_level_threshold" did you mean "
> min_compaction_threshold"  ? If so, why should I do that, what are the
> advantage/inconvenient of reducing this value ?
>
> Yes, min_compaction_threshold, my bad.
> If you have a wide row and delete a lot of values you will end up with a
> lot of tombstones. These may dramatically reduce the read performance until
> they are purged. Reducing the compaction threshold makes compaction happen
> more frequently.
>
> Looking at the doc I saw that: "max_compaction_threshold: Ignored in
> Cassandra 1.1 and later.". How to ensure that I'll always keep a small
> amount of SSTables then ?
>
> AFAIK it's not.
> There may be some confusion about the location of the settings in CLI vs
> CQL.
> Can you point to the docs.
>
> Cheers
>
>-
> Aaron Morton
> Freelance Cassandra Developer
> New Zealand
>
> @aaronmorton
> http://www.thelastpickle.com
>
> On 13/02/2013, at 10:14 PM, Alain RODRIGUEZ  wrote:
>
> Hi Aaron, once again thanks for this answer.
>
> "So is it possible to delete all the data inserted in some CF between 2
> dates or data older than 1 month ?"
>
> "No. "
>
> Why is there no way of deleting or getting data using the internal
> timestamp stored alongside of any inserted column (as described here:
> http://www.datastax.com/docs/1.1/ddl/column_family#standard-columns) ? Is
> that a feature that could possibly be developed one day ? It could
> be useful to perform delete of old data or to bring to a dev cluster just
> the last week of data for example.
>
> With "min_compaction_level_threshold" did you mean "
> min_compaction_threshold"  ? If so, why should I do that, what are the
> advantage/inconvenient of reducing this value ?
>
> Looking at the doc I saw that: "max_compaction_threshold: Ignored in
> Cassandra 1.1 and later.". How to ensure that I'll always keep a small
> amount of SSTables then ? Why is this deprecated ?
>
> Alain
>
>
> 2013/2/12 aaron morton 
>
>> So is it possible to delete all the data inserted in some CF between 2
>> dates or data older than 1 month ?
>>
>> No.
>>
>> You need to issue row level deletes. If you don't know the row key you'll
>> need to do range scans to locate them.
>>
>> If you are deleting parts of wide rows consider reducing the
>> min_compaction_level_threshold on the CF to 2
>>
>> Cheers
>>
>>
>>-
>> Aaron Morton
>> Freelance Cassandra Developer
>> New Zealand
>>
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 12/02/2013, at 4:21 AM, Alain RODRIGUEZ  wrote:
>>
>> Hi,
>>
>> I would like to know if there is a way to delete old/unused data easily ?
>>
>> I know about TTL but there are 2 limitations of TTL:
>>
>> - AFAIK, there is no TTL on counter columns
>> - TTL need to be defined at write time, so it's too late for data already
>> inserted.
>>
>> I also could use a standard "delete" but it seems inappropriate for such
>> a massive.
>>
>> In some cases, I don't know the row key and would like to delete all the
>> rows starting by, let's say, "1050#..."
>>
>> Even better, I understood that columns are always inserted in C* with
>> (name, value, timestamp). So is it possible to delete all the data inserted
>> in some CF between 2 dates or data older than 1 month ?
>>
>> Alain
>>
>>
>>
>
>

Is there any consolidated literature about Read/Write and Data Consistency in Cassandra ?

[Mateus Ferreira e Freitas]

Like articles with tests and conclusions about it, and such, and not like the 
documentation in DataStax, or the Cassandra Books.

Thank you.

  

Re: Is there any consolidated literature about Read/Write and Data Consistency in Cassandra ?

[Edward Capriolo]

Asking the question three times will not help getting it answered
faster. Furthermore, I believe no one has answered it because no one
understands what your asking.

Here is something with tests and conclusions and it is not written by
datastax or part of a book on cassandra.

http://pbs.cs.berkeley.edu/

On Sat, Feb 16, 2013 at 6:19 PM, Mateus Ferreira e Freitas
 wrote:
> Like articles with tests and conclusions about it, and such, and not like
> the documentation in DataStax, or the Cassandra Books.
>
> Thank you.
>

RE: Is there any consolidated literature about Read/Write and Data Consistency in Cassandra ?

[Mateus Ferreira e Freitas]

I'm sorry, it's because the mail is returning to me, and I thought it wasn't 
working. That link is similar to what I asked. I'm searching like,  for 
example, PhD,Dr,Ms dissertationsabout those topics in Cassandra; articles like 
http://www.cs.cornell.edu/projects/ladis2009/papers/lakshman-ladis2009.pdf, but 
specific to Read/Write operations and/or data consistency.Thank you.
 > Date: Sat, 16 Feb 2013 18:26:20 -0500
> Subject: Re: Is there any consolidated literature about Read/Write and Data 
> Consistency in Cassandra ?
> From: edlinuxg...@gmail.com
> To: user@cassandra.apache.org
> 
> Asking the question three times will not help getting it answered
> faster. Furthermore, I believe no one has answered it because no one
> understands what your asking.
> 
> Here is something with tests and conclusions and it is not written by
> datastax or part of a book on cassandra.
> 
> http://pbs.cs.berkeley.edu/
> 
> On Sat, Feb 16, 2013 at 6:19 PM, Mateus Ferreira e Freitas
>  wrote:
> > Like articles with tests and conclusions about it, and such, and not like
> > the documentation in DataStax, or the Cassandra Books.
> >
> > Thank you.
> >
  

Re: virtual nodes + map reduce = too many mappers

[Jonathan Ellis]

Wouldn't you have more than 256 splits anyway, given a normal amount of data?

(Default split size is 64k rows.)

On Fri, Feb 15, 2013 at 7:01 PM, Edward Capriolo  wrote:
> Seems like the hadoop Input format should combine the splits that are
> on the same node into the same map task, like Hadoop's
> CombinedInputFormat can. I am not sure who recommends vnodes as the
> default, because this is now the second problem (that I know of) of
> this class where vnodes has extra overhead,
> https://issues.apache.org/jira/browse/CASSANDRA-5161
>
> This seems to be the standard operating practice in c* now, enable
> things in the default configuration like new partitioners and newer
> features like vnodes, even though they are not heavily tested in the
> wild or well understood, then deal with fallout.
>
>
> On Fri, Feb 15, 2013 at 11:52 AM, cem  wrote:
>> Hi All,
>>
>> I have just started to use virtual nodes. I set the number of nodes to 256
>> as recommended.
>>
>> The problem that I have is when I run a mapreduce job it creates node * 256
>> mappers. It creates node * 256 splits. this effects the performance since
>> the range queries have a lot of overhead.
>>
>> Any suggestion to improve the performance? It seems like I need to lower the
>> number of virtual nodes.
>>
>> Best Regards,
>> Cem
>>
>>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder, http://www.datastax.com
@spyced

Re: virtual nodes + map reduce = too many mappers

[Edward Capriolo]

Split size does not have to equal block size.

http://hadoop.apache.org/docs/r1.1.1/api/org/apache/hadoop/mapred/lib/CombineFileInputFormat.html

An abstract InputFormat that returns CombineFileSplit's in
InputFormat.getSplits(JobConf, int) method. Splits are constructed
from the files under the input paths. A split cannot have files from
different pools. Each split returned may contain blocks from different
files. If a maxSplitSize is specified, then blocks on the same node
are combined to form a single split. Blocks that are left over are
then combined with other blocks in the same rack. If maxSplitSize is
not specified, then blocks from the same rack are combined in a single
split; no attempt is made to create node-local splits. If the
maxSplitSize is equal to the block size, then this class is similar to
the default spliting behaviour in Hadoop: each block is a locally
processed split. Subclasses implement
InputFormat.getRecordReader(InputSplit, JobConf, Reporter) to
construct RecordReader's for CombineFileSplit's.

Hive offers a CombinedHiveInputFormat

https://issues.apache.org/jira/browse/HIVE-74

Essentially Combined input formats rock hard. If you have a directory
with say 2000 files, you do not want 2000 splits, and then the
overhead of starting stopping 2000 mappers.

If you enable CombineInputFormat you can tune mapred.split.size and
the number of mappers is based (mostly) on the input size. This gives
jobs that would create too many map tasks way more throughput, and
stops them from monopolizing the map slots on the cluster.

It would seem like all the extra splits from the vnode change could be
combined back together.

On Sat, Feb 16, 2013 at 8:21 PM, Jonathan Ellis  wrote:
> Wouldn't you have more than 256 splits anyway, given a normal amount of data?
>
> (Default split size is 64k rows.)
>
> On Fri, Feb 15, 2013 at 7:01 PM, Edward Capriolo  
> wrote:
>> Seems like the hadoop Input format should combine the splits that are
>> on the same node into the same map task, like Hadoop's
>> CombinedInputFormat can. I am not sure who recommends vnodes as the
>> default, because this is now the second problem (that I know of) of
>> this class where vnodes has extra overhead,
>> https://issues.apache.org/jira/browse/CASSANDRA-5161
>>
>> This seems to be the standard operating practice in c* now, enable
>> things in the default configuration like new partitioners and newer
>> features like vnodes, even though they are not heavily tested in the
>> wild or well understood, then deal with fallout.
>>
>>
>> On Fri, Feb 15, 2013 at 11:52 AM, cem  wrote:
>>> Hi All,
>>>
>>> I have just started to use virtual nodes. I set the number of nodes to 256
>>> as recommended.
>>>
>>> The problem that I have is when I run a mapreduce job it creates node * 256
>>> mappers. It creates node * 256 splits. this effects the performance since
>>> the range queries have a lot of overhead.
>>>
>>> Any suggestion to improve the performance? It seems like I need to lower the
>>> number of virtual nodes.
>>>
>>> Best Regards,
>>> Cem
>>>
>>>
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder, http://www.datastax.com
> @spyced

NPE in running "ClientOnlyExample"

[Jain Rahul]

Hi All,

I am newbie to Cassandra and trying to run an example program 
"ClientOnlyExample"  taken from 
https://raw.github.com/apache/cassandra/cassandra-1.2/examples/client_only/src/ClientOnlyExample.java.
 But while executing  the program it gives me a null pointer exception. Can you 
guys please help me out what I am missing.

I am using Cassandra 1.2.1 version. I have pasted the logs at 
http://pastebin.com/pmADWCYe

Exception in thread "main" java.lang.NullPointerException
  at org.apache.cassandra.db.ColumnFamily.create(ColumnFamily.java:71)
  at org.apache.cassandra.db.ColumnFamily.create(ColumnFamily.java:66)
  at org.apache.cassandra.db.ColumnFamily.create(ColumnFamily.java:61)
  at org.apache.cassandra.db.ColumnFamily.create(ColumnFamily.java:56)
  at org.apache.cassandra.db.RowMutation.add(RowMutation.java:183)
  at org.apache.cassandra.db.RowMutation.add(RowMutation.java:204)
  at ClientOnlyExample.testWriting(ClientOnlyExample.java:78)
  at ClientOnlyExample.main(ClientOnlyExample.java:135)

Regards,
Rahul
This email and any attachments are confidential, and may be legally privileged 
and protected by copyright. If you are not the intended recipient dissemination 
or copying of this email is prohibited. If you have received this in error, 
please notify the sender by replying by email and then delete the email 
completely from your system. Any views or opinions are solely those of the 
sender. This communication is not intended to form a binding contract unless 
expressly indicated to the contrary and properly authorised. Any actions taken 
on the basis of this email are at the recipient's own risk.