Hi Zhiliang, Would something like this work?
val rdd2 = rdd1.sliding(2).map(v => v(1) - v(0)) -sujit On Mon, Sep 21, 2015 at 7:58 AM, Zhiliang Zhu <zchl.j...@yahoo.com.invalid> wrote: > Hi Romi, > > Thanks very much for your kind help comment~~ > > In fact there is some valid backgroud of the application, it is about R > data analysis. > ... > #fund_nav_daily is a M X N (or M X 1) matrix or data.frame, col is each > daily fund return, row is the daily date > #fund_return_daily needs to count the each fund's daily return subtracted > the previous day's return > fund_return_daily <- diff(log(fund_nav_daily)) > > #the first row would be all 0, since there is no previous row ahead first > row > fund_return_daily <- rbind(matrix(0,ncol = ncol(fund_return_daily)), > fund_return_daily) > ... > > I need to exactly code the R program by way of spark, then RDD/DataFrame > is used to replace R data.frame, > however, I just found that it is VERY MUCH diffcult to make the spark > program to flexibly descript & transform R backgroud applications. > I think I have seriously lost myself into risk about this... > > Would you help direct me some about the above coding issue... and my risk > about practice in spark/R application... > > I must show all my sincere thanks torwards your kind help. > > P.S. currently sparkR in spark 1.4.1 , there is many bug in the API > createDataFrame/except/unionAll, and it seems > that spark Java has more functions than sparkR. > Also, no specific R regression algorithmn is including in sparkR . > > Best Regards, > Zhiliang > > > On Monday, September 21, 2015 7:36 PM, Romi Kuntsman <r...@totango.com> > wrote: > > > RDD is a set of data rows (in your case numbers), there is no meaning for > the order of the items. > What exactly are you trying to accomplish? > > *Romi Kuntsman*, *Big Data Engineer* > http://www.totango.com > > On Mon, Sep 21, 2015 at 2:29 PM, Zhiliang Zhu <zchl.j...@yahoo.com.invalid > > wrote: > > Dear , > > I have took lots of days to think into this issue, however, without any > success... > I shall appreciate your all kind help. > > There is an RDD<int> rdd1, I would like get a new RDD<int> rdd2, each row > in rdd2[ i ] = rdd1[ i ] - rdd[i - 1] . > What kinds of API or function would I use... > > > Thanks very much! > John > > > > >
