Re: RDD to Multiple Tables SparkSQL

2014-10-28 Thread critikaled
Hi oliver, thanks for the answer I don't have the information of all keys before hand, the reason i want to have multiple tables is that based on my information on known key I will apply different queries get the results for that particular key I don't want to touch the unkown ones I'll save that f

Re: RDD to Multiple Tables SparkSQL

2014-10-21 Thread Olivier Girardot
If you already know your keys the best way would be to "extract" one RDD per key (it would not bring the content back to the master and you can take advantage of the caching features) and then execute a registerTempTable by Key. But I'm guessing, you don't know the keys in advance, and in this cas