Re: Maximum Size of Reference Look Up Table in Spark

2016-07-15 Thread Saravanan Subramanian
/@jaceklaskowski/ Mastering Apache Spark http://bit.ly/mastering-apache-spark Follow me at https://twitter.com/jaceklaskowski On Thu, Jul 14, 2016 at 11:32 PM, Saravanan Subramanian wrote: > Hello All, > > I am in the middle of designing real time data enhancement services using > spa

Maximum Size of Reference Look Up Table in Spark

2016-07-14 Thread Saravanan Subramanian
Hello All, I am in the middle of designing real time data enhancement services using spark streaming.  As part of this, I have to look up some reference data while processing the incoming stream. I have below questions: 1) what is the maximum size of look up table / variable can be stored as Bro