Can you try caching the individual dataframes and then union them? It may save you time.
Thanks Deepak On Wed, Nov 16, 2016 at 12:35 PM, Devi P.V <devip2...@gmail.com> wrote: > Hi all, > > I have 4 data frames with three columns, > > client_id,product_id,interest > > I want to combine these 4 dataframes into one dataframe.I used union like > following > > df1.union(df2).union(df3).union(df4) > > But it is time consuming for bigdata.what is the optimized way for doing > this using spark 2.0 & scala > > > Thanks > -- Thanks Deepak www.bigdatabig.com www.keosha.net
