I agree with Jorn et al on this. An alternative approach would be best. as a 24 hour operation sounds like a classic batch job more suitable for later reporting. This happens all the time in RDBMS.
As I understand and within Spark the sliding interval can only be used after that window length (in this case 24 hours) has elapsed. You might as well use normal storage for it. It may be slower but would be far more manageable. Otherwise use other suggestions I made. HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com On 9 May 2016 at 15:15, firemonk9 <dhiraj.peech...@gmail.com> wrote: > I have not come across official docs in this regard how ever if you use 24 > hour window size, you will need to have memory big enough to fit the stream > data for 24 hours. Usually memory is the limiting factor for the window > size. > > Dhiraj Peechara > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/How-big-the-spark-stream-window-could-be-tp26899p26903.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
