Hi, thank you I would prefer to leave writing-to-disk as a last resort. Is it a last resort?
Saif From: Ted Yu [mailto:[email protected]] Sent: Friday, October 02, 2015 3:54 PM To: Ellafi, Saif A. Cc: user Subject: Re: how to broadcast huge lookup table? Have you considered using external storage such as hbase for storing the look up table ? Cheers On Fri, Oct 2, 2015 at 11:50 AM, <[email protected]<mailto:[email protected]>> wrote: I tried broadcasting a key-value rdd, but then I cannot perform any rdd-actions inside a map/foreach function of another rdd. any tips? If going into scala collections I end up with huge memory bottlenecks. Saif
