Re: SQLContext within foreachRDD

2015-10-12 Thread Daniel Haviv
Just wanted to make sure. Thanks. Daniel On Mon, Oct 12, 2015 at 1:07 PM, Adrian Tanase wrote: > Not really, unless you’re doing something wrong (e.g. Call collect or > similar). > > In the foreach loop you’re typically registering a temp table, by > converting an RDD to data frame. All the sub

Re: SQLContext within foreachRDD

2015-10-12 Thread Adrian Tanase
Not really, unless you’re doing something wrong (e.g. Call collect or similar). In the foreach loop you’re typically registering a temp table, by converting an RDD to data frame. All the subsequent queries are executed in parallel on the workers. I haven’t built production apps with this patter