RE: [SQL] Self join with ArrayType columns problems

2015-01-27 Thread Cheng, Hao
: Tuesday, January 27, 2015 12:05 AM To: Dean Wampler Cc: Pierre B; user@spark.apache.org; Cheng Hao Subject: Re: [SQL] Self join with ArrayType columns problems It seems likely that there is some sort of bug related to the reuse of array objects that are returned by UDFs. Can you open a JIRA? I&#x

Re: [SQL] Self join with ArrayType columns problems

2015-01-26 Thread Michael Armbrust
It seems likely that there is some sort of bug related to the reuse of array objects that are returned by UDFs. Can you open a JIRA? I'll also note that the sql method on HiveContext does run HiveQL (configured by spark.sql.dialect) and the hql method has been deprecated since 1.1 (and will proba

Re: [SQL] Self join with ArrayType columns problems

2015-01-26 Thread Dean Wampler
You are creating a HiveContext, then using the sql method instead of hql. Is that deliberate? The code doesn't work if you replace HiveContext with SQLContext. Lots of exceptions are thrown, but I don't have time to investigate now. dean Dean Wampler, Ph.D. Author: Programming Scala, 2nd Edition