Re: SQL UDF problem (with re to types)

2016-01-17 Thread Ted Yu
While reading some book on Java 8, I saw a reference to the following w.r.t. declaration-site variance : https://bugs.openjdk.java.net/browse/JDK-8043488 The above reportedly targets Java 9. FYI On Thu, Jan 14, 2016 at 12:33 PM, Michael Armbrust wrote: > I don't believe that Java 8 got rid of

Re: SQL UDF problem (with re to types)

2016-01-14 Thread Michael Armbrust
I don't believe that Java 8 got rid of erasure. In fact I think its actually worse when you use Java 8 lambdas. On Thu, Jan 14, 2016 at 10:54 AM, Raghu Ganti wrote: > Would this go away if the Spark source was compiled against Java 1.8 > (since the problem of type erasure is solved through prope

Re: SQL UDF problem (with re to types)

2016-01-14 Thread Raghu Ganti
Would this go away if the Spark source was compiled against Java 1.8 (since the problem of type erasure is solved through proper generics implementation in Java 1.8). On Thu, Jan 14, 2016 at 1:42 PM, Michael Armbrust wrote: > We automatically convert types for UDFs defined in Scala, but we can't

Re: SQL UDF problem (with re to types)

2016-01-14 Thread Michael Armbrust
We automatically convert types for UDFs defined in Scala, but we can't do it in Java because the types are erased by the compiler. If you want to use double you should cast before calling the UDF. On Wed, Jan 13, 2016 at 8:10 PM, Raghu Ganti wrote: > So, when I try BigDecimal, it works. But, sh

Re: SQL UDF problem (with re to types)

2016-01-13 Thread Raghu Ganti
So, when I try BigDecimal, it works. But, should it not parse based on what the UDF defines? Am I missing something here? On Wed, Jan 13, 2016 at 4:57 PM, Ted Yu wrote: > Please take a look > at sql/hive/src/test/java/org/apache/spark/sql/hive/aggregate/MyDoubleSum.java > which shows a UserDefin

Re: SQL UDF problem (with re to types)

2016-01-13 Thread Ted Yu
Please take a look at sql/hive/src/test/java/org/apache/spark/sql/hive/aggregate/MyDoubleSum.java which shows a UserDefinedAggregateFunction that works on DoubleType column. sql/hive/src/test/java/org/apache/spark/sql/hive/JavaDataFrameSuite.java shows how it is registered. Cheers On Wed, Jan 13

Re: SQL UDF problem (with re to types)

2016-01-13 Thread Ted Yu
Looks like BigDecimal was passed to your call() method. Can you modify your udf to see if using BigDecimal works ? Cheers On Wed, Jan 13, 2016 at 11:58 AM, raghukiran wrote: > While registering and using SQL UDFs, I am running into the following > problem: > > UDF registered: > > ctx.udf().reg