Re: AnalysisException: Trouble using select() to append multiple columns

2021-12-24 Thread Sean Owen
t;>> Thanks Nicholas >>>> >>>> >>>> >>>> Andy >>>> >>>> >>>> >>>> *From: *Nicholas Gustafson >>>> *Date: *Friday, December 17, 2021 at 6:12 PM >>>> *To: *Andrew Davidson >>>

Re: AnalysisException: Trouble using select() to append multiple columns

2021-12-24 Thread Gourav Sengupta
Davidson >> wrote: >> >>> Thanks Nicholas >>> >>> >>> >>> Andy >>> >>> >>> >>> *From: *Nicholas Gustafson >>> *Date: *Friday, December 17, 2021 at 6:12 PM >>> *To: *Andrew Davidson >&g

Re: AnalysisException: Trouble using select() to append multiple columns

2021-12-24 Thread Sean Owen
ustafson >> *Date: *Friday, December 17, 2021 at 6:12 PM >> *To: *Andrew Davidson >> *Cc: *"user@spark.apache.org" >> *Subject: *Re: AnalysisException: Trouble using select() to append >> multiple columns >> >> >> >> Since df1 and df2 are dif

Re: AnalysisException: Trouble using select() to append multiple columns

2021-12-24 Thread Gourav Sengupta
las > > > > Andy > > > > *From: *Nicholas Gustafson > *Date: *Friday, December 17, 2021 at 6:12 PM > *To: *Andrew Davidson > *Cc: *"user@spark.apache.org" > *Subject: *Re: AnalysisException: Trouble using select() to append > multiple columns >

Re: AnalysisException: Trouble using select() to append multiple columns

2021-12-18 Thread Andrew Davidson
Thanks Nicholas Andy From: Nicholas Gustafson Date: Friday, December 17, 2021 at 6:12 PM To: Andrew Davidson Cc: "user@spark.apache.org" Subject: Re: AnalysisException: Trouble using select() to append multiple columns Since df1 and df2 are different DataFrames, you will need to

Re: AnalysisException: Trouble using select() to append multiple columns

2021-12-17 Thread Nicholas Gustafson
Since df1 and df2 are different DataFrames, you will need to use a join. For example: df1.join(df2.selectExpr(“Name”, “NumReads as ctrl_2”), on=[“Name”]) > On Dec 17, 2021, at 16:25, Andrew Davidson wrote: > >  > Hi I am a newbie > > I have 16,000 data files, all files have the same number o