For the 2nd use case, can you save the result for first 29 days, then just get the last day result and add yourself ? This can be done outside of spark. Does that work for you
Sent from my iPad > On Nov 25, 2015, at 9:46 PM, Sachith Withana <swsach...@gmail.com> wrote: > > Hi folks! > > I'm wondering if Sparks supports or hopes to support incremental data > analysis. > > There are few use cases that prompted me to wonder. > > ex: If we need to summarize last 30 days worth of data everyday, > > 1. Does Spark support time range based query execution ? > select * from foo where timestamp in last30days > > 2. Can we reuse the previously executed data? > ex: use the sum of the last 29 days and add the last days worth of data? > > The second usecase is very tempting as it would improve the performance > drastically. > > Any suggestions would be greatly appreciated. > > -- > Thanks, > Sachith Withana >
