Re: Help with Spark Streaming

I guess, maybe you don’t need invoke reduceByKey() after mapToPair, because updateStateByKey had covered it. For your reference, here is a sample written by scala using text file stream instead of socket as below: object LocalStatefulWordCount extends App { val sparkConf = new SparkConf().setA

Using Spark Streaming to listen to HDFS directory and handle different files by file name

StreamContext provide the similar function to listen to the incoming files on HDFS? So that I can handle different files by file name on Spark Streaming. -- ZhangYi (张逸) Developer tel: 15023157626 blog: agiledon.github.com weibo: tw张逸 Sent with Sparrow (http://www.sparrowmailapp.com/?sig)

Is any idea on architecture based on Spark + Spray + Akka

us, and we can't find any best practice via google. In our opinion, event-driven architecture is good choice for our project maybe. However, more idea is welcome. Thanks. -- ZhangYi (张逸) Developer tel: 15023157626 blog: agiledon.github.com (http://agiledon.github.com) weibo: tw张逸

Re: My talk on "Spark: The Next Top (Compute) Model"

Very Useful material. Currently, I am trying to persuade my client choose Spark instead of Hadoop MapReduce. Your slide give me more evidence to support my opinion. -- ZhangYi (张逸) Developer tel: 15023157626 blog: agiledon.github.com weibo: tw张逸 Sent with Sparrow (http