At a high-level, the suggestion sounds good to me. However regarding code, its best to submit a Pull Request on Spark github page for community reviewing. You will find more information here. https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark
On Tue, Sep 23, 2014 at 10:11 PM, Soumitra Kumar <kumar.soumi...@gmail.com> wrote: > I thought I did a good job ;-) > > OK, so what is the best way to initialize updateStateByKey operation? I > have counts from previous spark-submit, and want to load that in next > spark-submit job. > > ----- Original Message ----- > From: "Soumitra Kumar" <kumar.soumi...@gmail.com> > To: "spark users" <user@spark.apache.org> > Sent: Sunday, September 21, 2014 10:43:01 AM > Subject: How to initialize updateStateByKey operation > > I started with StatefulNetworkWordCount to have a running count of words > seen. > > I have a file 'stored.count' which contains the word counts. > > $ cat stored.count > a 1 > b 2 > > I want to initialize StatefulNetworkWordCount with the values in > 'stored.count' file, how do I do that? > > I looked at the paper 'EECS-2012-259.pdf' Matei et al, in figure 2, it > would be useful to have an initial RDD feeding into 'counts' at 't = 1', as > below. > > initial > | > t = 1: pageView -> ones -> counts > | > t = 2: pageView -> ones -> counts > ... > > I have also attached the modified figure 2 of > http://www.eecs.berkeley.edu/Pubs/TechRpts/2012/EECS-2012-259.pdf . > > I managed to hack Spark code to achieve this, and attaching the modified > files. > > Essentially, I added an argument 'initial : RDD [(K, S)]' to > updateStateByKey method, as > def updateStateByKey[S: ClassTag]( > initial : RDD [(K, S)], > updateFunc: (Seq[V], Option[S]) => Option[S], > partitioner: Partitioner > ): DStream[(K, S)] > > If it sounds interesting for larger crowd I would be happy to cleanup the > code, and volunteer to push into the code. I don't know the procedure to > that though. > > -Soumitra. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
