“counties”);
> …
> And then doing the join.
>
>
> From: Raghu Ganti [mailto:raghuki...@gmail.com]
> Sent: Thursday, January 28, 2016 3:06 AM
> To: Ted Yu; Дмитро Попович
> Cc: user
> Subject: Re: Spark SQL joins taking too long
>
> The problem is with the way S
The problem is with the way Spark query plan is being created, IMO, what
was happening before is that the order of the tables mattered and when the
larger table is given first, it took a very long time (~53mins to
complete). I changed the order of the tables with the smaller one first
(including re
Yes, the SomeUDF is Contains, shape is a UDT that maps a custom geometry type
to sql binary type.
Custom geometry type is a Java class. Please let me know if you need further
info.
Regards
Raghu
> On Jan 26, 2016, at 17:13, Ted Yu wrote:
>
> What's the type of shape column ?
>
> Can you dis
Java class. Scala case classes implement the scala.Product
> trait, which Catalyst is looking for.
>
>
> Thanks,
>
> Andy.
>
> --
>
> Andy Grove
> Chief Architect
> AgilData - Simple Streaming SQL that Scales
> www.agildata.com
>
>
> On Wed, Jan
Is it not internal to the Catalyst implementation? I should not be
modifying the Spark source to get things to work, do I? :-)
On Wed, Jan 20, 2016 at 12:21 PM, Raghu Ganti wrote:
> Case classes where?
>
> On Wed, Jan 20, 2016 at 12:21 PM, Andy Grove
> wrote:
>
>> Honestly
gilData - Simple Streaming SQL that Scales
> www.agildata.com
>
>
> On Wed, Jan 20, 2016 at 10:19 AM, Raghu Ganti
> wrote:
>
>> Thanks for your reply, Andy.
>>
>> Yes, that is what I concluded based on the Stack trace. The problem is
>> stemming from Java impleme
Thanks for your reply, Andy.
Yes, that is what I concluded based on the Stack trace. The problem is
stemming from Java implementation of generics, but I thought this will go
away if you compiled against Java 1.8, which solves the issues of proper
generic implementation.
Any ideas?
Also, are you
ean SQLContext.
> From the doc for SQLContext#createDataFrame():
>
>* dataFrame.registerTempTable("people")
>* sqlContext.sql("select name from people").collect.foreach(println)
>
> If you want to persist table externally, you need Hive, etc
>
> Regards
&g
This requires Hive to be installed and uses HiveContext, right?
What is the SparkSQLContext useful for?
On Mon, Jan 18, 2016 at 1:27 PM, Ted Yu wrote:
> Please take a look
> at sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveDataFrameSuite.scala
>
> On Mon, Jan 18, 2016 at 9:57 AM, raghuk
Btw, Thanks a lot for all your quick responses - it is very useful and
definitely appreciate it :-)
On Mon, Jan 18, 2016 at 1:28 PM, Raghu Ganti wrote:
> This requires Hive to be installed and uses HiveContext, right?
>
> What is the SparkSQLContext useful for?
>
> On Mon, Jan 1
we can't do
> it in Java because the types are erased by the compiler. If you want to
> use double you should cast before calling the UDF.
>
> On Wed, Jan 13, 2016 at 8:10 PM, Raghu Ganti wrote:
>
>> So, when I try BigDecimal, it works. But, should it not parse based on
So, when I try BigDecimal, it works. But, should it not parse based on what
the UDF defines? Am I missing something here?
On Wed, Jan 13, 2016 at 4:57 PM, Ted Yu wrote:
> Please take a look
> at sql/hive/src/test/java/org/apache/spark/sql/hive/aggregate/MyDoubleSum.java
> which shows a UserDefin
12 matches
Mail list logo
