Re: mysterious spark.sql.utils.AnalysisException Union in spark 3.3.2, but not seen in 3.4.0+

Hi Srivastan, Ground investigation 1. Does this union explicitly exist in your code? If not, where are the 7 and 6 column counting coming from? 2. On 3.3.1 have you looked at spark UI and the relevant dag diagram 3. Check query execution plan using explain() functionality 4. Can yo

mysterious spark.sql.utils.AnalysisException Union in spark 3.3.2, but not seen in 3.4.0+

Hello Users, I have been seeing some weird issues when I upgraded my EMR setup to 6.11 (which uses spark 3.3.2) , the call stack seems to point to a code location where there is no explicit union, also I have unionByName everywhere in the codebase with allowMissingColumns set to

Re:unresolved operator union in Spark 1.6

You can upgrade your Spark to version 2.x. At 2019-08-20 19:16:50, "马阳阳" wrote: Hi all, We encountered the following problem when using Spark 1.6. The following SQL: ``` SELECT a,b,c FROM A UNION ALL SELECT a,b,c FROM B ``` If A and B are two views created with SQLContext.registerDat

unresolved operator union in Spark 1.6

Hi all, We encountered the following problem when using Spark 1.6. The following SQL: ``` SELECT a,b,c FROM A UNION ALL SELECT a,b,c FROM B ``` If A and B are two views created with SQLContext.registerDataFrameAsTable(), Spark will fail with “unresolved operator Union”. But if A and B are s

Unresolved operator Union in Spark 1.6

Hi all, We encountered the following problem when using Spark 1.6. The following SQL: ``` SELECT a,b,c FROM A UNION ALL SELECT a,b,c FROM B ``` If A and B are two views created with SQLContext.registerDataFrameAsTable(), Spark will fail with “unresolved operator Union”. But if A and B

Re: Union in Spark

The configuration is 16GB ram and 1TB HD. have a single node Spark cluster. Even after setting driver memory to 5g and executor memory to 3g, I get this error. The size of the data set is 350 KB and the set that it works well is hardly few KBs. On Mon, Feb 2, 2015 at 1:18 PM, Arush Kharbanda wrot

Re: Union in Spark

Hi Deep, What is your configuration and what is the size of the 2 data sets? Thanks Arush On Mon, Feb 2, 2015 at 11:56 AM, Deep Pradhan wrote: > I did not check the console because once the job starts I cannot run > anything else and have to force shutdown the system. I commented parts of > co

Re: Union in Spark

I did not check the console because once the job starts I cannot run anything else and have to force shutdown the system. I commented parts of codes and I tested. I doubt it is because of union. So, I want to change it to something else and see if the problem persists. Thank you On Mon, Feb 2, 20

Re: Union in Spark

Hi Deep, How do you know the cluster is not responsive because of "Union"? Did you check the spark web console? Best Regards, Jerry On Mon, Feb 2, 2015 at 1:21 AM, Deep Pradhan wrote: > The cluster hangs. > > On Mon, Feb 2, 2015 at 11:25 AM, Jerry Lam wrote: > >> Hi Deep, >> >> what do you

Re: Union in Spark

The cluster hangs. On Mon, Feb 2, 2015 at 11:25 AM, Jerry Lam wrote: > Hi Deep, > > what do you mean by stuck? > > Jerry > > > On Mon, Feb 2, 2015 at 12:44 AM, Deep Pradhan > wrote: > >> Hi, >> Is there any better operation than Union. I am using union and the >> cluster is getting stuck with a

Re: Union in Spark

Hi Deep, what do you mean by stuck? Jerry On Mon, Feb 2, 2015 at 12:44 AM, Deep Pradhan wrote: > Hi, > Is there any better operation than Union. I am using union and the cluster > is getting stuck with a large data set. > > Thank you >

Union in Spark

Hi, Is there any better operation than Union. I am using union and the cluster is getting stuck with a large data set. Thank you