Pardon me.
I didn't read your previous response clearly.
I will try to reproduce the compilation error on master branch.
Right now, I have some other high priority task on hand.
BTW I was looking at SPARK-10671
FYI
On Tue, Oct 13, 2015 at 1:42 AM, Umesh Kacha <umesh.ka...@gmail.com> wrote:
> Hi Ted if fix went after 1.5.1 release then how come it's working with
> 1.5.1 binary in spark-shell.
> On Oct 13, 2015 1:32 PM, "Ted Yu" <yuzhih...@gmail.com> wrote:
>
>> Looks like the fix went in after 1.5.1 was released.
>>
>> You may verify using master branch build.
>>
>> Cheers
>>
>> On Oct 13, 2015, at 12:21 AM, Umesh Kacha <umesh.ka...@gmail.com> wrote:
>>
>> Hi Ted, thanks much I tried using percentile_approx in Spark-shell like
>> you mentioned it works using 1.5.1 but it doesn't compile in Java using
>> 1.5.1 maven libraries it still complains same that callUdf can have string
>> and column types only. Please guide.
>> On Oct 13, 2015 12:34 AM, "Ted Yu" <yuzhih...@gmail.com> wrote:
>>
>>> SQL context available as sqlContext.
>>>
>>> scala> val df = Seq(("id1", 1), ("id2", 4), ("id3", 5)).toDF("id",
>>> "value")
>>> df: org.apache.spark.sql.DataFrame = [id: string, value: int]
>>>
>>> scala> df.select(callUDF("percentile_approx",col("value"),
>>> lit(0.25))).show()
>>> +------------------------------+
>>> |'percentile_approx(value,0.25)|
>>> +------------------------------+
>>> | 1.0|
>>> +------------------------------+
>>>
>>> Can you upgrade to 1.5.1 ?
>>>
>>> Cheers
>>>
>>> On Mon, Oct 12, 2015 at 11:55 AM, Umesh Kacha <umesh.ka...@gmail.com>
>>> wrote:
>>>
>>>> Sorry forgot to tell that I am using Spark 1.4.1 as callUdf is
>>>> available in Spark 1.4.0 as per JAvadocx
>>>>
>>>> On Tue, Oct 13, 2015 at 12:22 AM, Umesh Kacha <umesh.ka...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Ted thanks much for the detailed answer and appreciate your
>>>>> efforts. Do we need to register Hive UDFs?
>>>>>
>>>>> sqlContext.udf.register("percentile_approx");???//is it valid?
>>>>>
>>>>> I am calling Hive UDF percentile_approx in the following manner which
>>>>> gives compilation error
>>>>>
>>>>> df.select("col1").groupby("col1").agg(callUdf("percentile_approx",col("col1"),lit(0.25)));//compile
>>>>> error
>>>>>
>>>>> //compile error because callUdf() takes String and Column* as
>>>>> arguments.
>>>>>
>>>>> Please guide. Thanks much.
>>>>>
>>>>> On Mon, Oct 12, 2015 at 11:44 PM, Ted Yu <yuzhih...@gmail.com> wrote:
>>>>>
>>>>>> Using spark-shell, I did the following exercise (master branch) :
>>>>>>
>>>>>>
>>>>>> SQL context available as sqlContext.
>>>>>>
>>>>>> scala> val df = Seq(("id1", 1), ("id2", 4), ("id3", 5)).toDF("id",
>>>>>> "value")
>>>>>> df: org.apache.spark.sql.DataFrame = [id: string, value: int]
>>>>>>
>>>>>> scala> sqlContext.udf.register("simpleUDF", (v: Int, cnst: Int) => v
>>>>>> * v + cnst)
>>>>>> res0: org.apache.spark.sql.UserDefinedFunction =
>>>>>> UserDefinedFunction(<function2>,IntegerType,List())
>>>>>>
>>>>>> scala> df.select($"id", callUDF("simpleUDF", $"value",
>>>>>> lit(25))).show()
>>>>>> +---+--------------------+
>>>>>> | id|'simpleUDF(value,25)|
>>>>>> +---+--------------------+
>>>>>> |id1| 26|
>>>>>> |id2| 41|
>>>>>> |id3| 50|
>>>>>> +---+--------------------+
>>>>>>
>>>>>> Which Spark release are you using ?
>>>>>>
>>>>>> Can you pastebin the full stack trace where you got the error ?
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>> On Fri, Oct 9, 2015 at 1:09 PM, Umesh Kacha <umesh.ka...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I have a doubt Michael I tried to use callUDF in the following code
>>>>>>> it does not work.
>>>>>>>
>>>>>>> sourceFrame.agg(callUdf("percentile_approx",col("myCol"),lit(0.25)))
>>>>>>>
>>>>>>> Above code does not compile because callUdf() takes only two
>>>>>>> arguments function name in String and Column class type. Please guide.
>>>>>>>
>>>>>>> On Sat, Oct 10, 2015 at 1:29 AM, Umesh Kacha <umesh.ka...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> thanks much Michael let me try.
>>>>>>>>
>>>>>>>> On Sat, Oct 10, 2015 at 1:20 AM, Michael Armbrust <
>>>>>>>> mich...@databricks.com> wrote:
>>>>>>>>
>>>>>>>>> This is confusing because I made a typo...
>>>>>>>>>
>>>>>>>>> callUDF("percentile_approx", col("mycol"), lit(0.25))
>>>>>>>>>
>>>>>>>>> The first argument is the name of the UDF, all other arguments
>>>>>>>>> need to be columns that are passed in as arguments. lit is just
>>>>>>>>> saying to
>>>>>>>>> make a literal column that always has the value 0.25.
>>>>>>>>>
>>>>>>>>> On Fri, Oct 9, 2015 at 12:16 PM, <saif.a.ell...@wellsfargo.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Yes but I mean, this is rather curious. How is def
>>>>>>>>>> lit(literal:Any) --> becomes a percentile function lit(25)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks for clarification
>>>>>>>>>>
>>>>>>>>>> Saif
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> *From:* Umesh Kacha [mailto:umesh.ka...@gmail.com]
>>>>>>>>>> *Sent:* Friday, October 09, 2015 4:10 PM
>>>>>>>>>> *To:* Ellafi, Saif A.
>>>>>>>>>> *Cc:* Michael Armbrust; user
>>>>>>>>>>
>>>>>>>>>> *Subject:* Re: How to calculate percentile of a column of
>>>>>>>>>> DataFrame?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I found it in 1.3 documentation lit says something else not
>>>>>>>>>> percent
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> public static Column
>>>>>>>>>> <https://spark.apache.org/docs/1.3.1/api/java/org/apache/spark/sql/Column.html>
>>>>>>>>>> lit(Object literal)
>>>>>>>>>>
>>>>>>>>>> Creates a Column
>>>>>>>>>> <https://spark.apache.org/docs/1.3.1/api/java/org/apache/spark/sql/Column.html>
>>>>>>>>>> of
>>>>>>>>>> literal value.
>>>>>>>>>>
>>>>>>>>>> The passed in object is returned directly if it is already a
>>>>>>>>>> Column
>>>>>>>>>> <https://spark.apache.org/docs/1.3.1/api/java/org/apache/spark/sql/Column.html>.
>>>>>>>>>> If the object is a Scala Symbol, it is converted into a Column
>>>>>>>>>> <https://spark.apache.org/docs/1.3.1/api/java/org/apache/spark/sql/Column.html>
>>>>>>>>>> also.
>>>>>>>>>> Otherwise, a new Column
>>>>>>>>>> <https://spark.apache.org/docs/1.3.1/api/java/org/apache/spark/sql/Column.html>
>>>>>>>>>> is
>>>>>>>>>> created to represent the literal value.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Sat, Oct 10, 2015 at 12:39 AM, <saif.a.ell...@wellsfargo.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Where can we find other available functions such as lit() ? I
>>>>>>>>>> can’t find lit in the api.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> *From:* Michael Armbrust [mailto:mich...@databricks.com]
>>>>>>>>>> *Sent:* Friday, October 09, 2015 4:04 PM
>>>>>>>>>> *To:* unk1102
>>>>>>>>>> *Cc:* user
>>>>>>>>>> *Subject:* Re: How to calculate percentile of a column of
>>>>>>>>>> DataFrame?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> You can use callUDF(col("mycol"), lit(0.25)) to call hive UDFs
>>>>>>>>>> from dataframes.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, Oct 9, 2015 at 12:01 PM, unk1102 <umesh.ka...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi how to calculate percentile of a column in a DataFrame? I cant
>>>>>>>>>> find any
>>>>>>>>>> percentile_approx function in Spark aggregation functions. For
>>>>>>>>>> e.g. in Hive
>>>>>>>>>> we have percentile_approx and we can use it in the following way
>>>>>>>>>>
>>>>>>>>>> hiveContext.sql("select percentile_approx("mycol",0.25) from
>>>>>>>>>> myTable);
>>>>>>>>>>
>>>>>>>>>> I can see ntile function but not sure how it is gonna give
>>>>>>>>>> results same as
>>>>>>>>>> above query please guide.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> View this message in context:
>>>>>>>>>> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-calculate-percentile-of-a-column-of-DataFrame-tp25000.html
>>>>>>>>>> Sent from the Apache Spark User List mailing list archive at
>>>>>>>>>> Nabble.com <http://nabble.com>.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>>>>>>>>>> For additional commands, e-mail: user-h...@spark.apache.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
