Spark streaming kafka cost long time at "take at DStream.scala:586"

2014-07-04 Thread xiemeilong
I am using : kafka 0.8.1 spark-streaming-kafka_2.10-0.9.0-cdh5.0.2 My analysis is simple, so I confuse why it cost so long time at "take at DStream.scala:586", it cost 2 to 8 minutes or longer .I don't know how to find the reason. Hoping your help. Sorry for my poor english. -- View this m

Re: combinebykey throw classcastexception

2014-05-20 Thread xiemeilong
This issue is turned out cased by version mismatch between driver(0.9.1) and server(0.9.0-cdh5.0.1) just now. Other function works fine but combinebykey before. Thank you very much for your reply. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/combinebyke

combinebykey throw classcastexception

2014-05-19 Thread xiemeilong
I am using CDH5 on a three machines cluster. map data from hbase as (string, V) pair , then call combineByKey like this: .combineByKey[C]( (v:V)=>new C(v), //this line throw java.lang.ClassCastException: C cannot be cast to V (v:C,v:V)=>C, (c1:C,c2:C)=>C) I am very confu