Thanks great Dhaval.
scala> import java.text.SimpleDateFormat
import java.text.SimpleDateFormat
scala>
scala> import java.sql.Date
import java.sql.Date
scala>
scala> import scala.util.{Try, Success, Failure}
import scala.util.{Try, Success, Failure}
scala> val toDate = udf{(out:String, form: Strin
Hi Mich,
You can try this:
val toDate = udf{(out:String, form: String) => {
val format = new SimpleDateFormat(s"$form");
Try(new Date(format.parse(out.toString()).getTime))match {
case Success(t) => Some(t)
case Failure(_) => None
}}};
Usage: src = src.withColumn(s"$columnName", toDate(src(s"$co
Hi Ted,
I decided to take a short cut here. I created the map leaving date as it is
(p(1)) below
def CleanupCurrency (word : String) : Double = {
return word.toString.substring(1).replace(",", "").toDouble
}
sqlContext.udf.register("CleanupCurrency", CleanupCurrency(_:String))
val a = df.filter(c
Mich:
Please take a look at:
sql/core/src/test/scala/org/apache/spark/sql/DateFunctionsSuite.scala
test("function to_date") {
Remember to:
import org.apache.spark.sql.functions._
On Fri, Mar 25, 2016 at 7:59 AM, Mich Talebzadeh
wrote:
> This works with sql
>
> sqltext = """
> INSERT INTO TABLE
This works with sql
sqltext = """
INSERT INTO TABLE t14
SELECT
INVOICENUMBER
,
TO_DATE(FROM_UNIXTIME(UNIX_TIMESTAMP(paymentdate,'dd/MM/'),'-MM-dd'))
AS paymentdate
, NET
, VAT
, TOTAL
FROM tmp
"""
sql(sqltext)
but not in UDF. I want to convert i
Do you mind showing body of TO_DATE() ?
Thanks
On Fri, Mar 25, 2016 at 7:38 AM, Ted Yu wrote:
> Looks like you forgot an import for Date.
>
> FYI
>
> On Fri, Mar 25, 2016 at 7:36 AM, Mich Talebzadeh <
> mich.talebza...@gmail.com> wrote:
>
>>
>>
>> Hi,
>>
>> writing a UDF to convert a string in
Looks like you forgot an import for Date.
FYI
On Fri, Mar 25, 2016 at 7:36 AM, Mich Talebzadeh
wrote:
>
>
> Hi,
>
> writing a UDF to convert a string into Date
>
> def ChangeDate(word : String) : Date = {
> | return
> TO_DATE(FROM_UNIXTIME(UNIX_TIMESTAMP(word),"dd/MM/"),"-MM-dd")
Hi,
writing a UDF to convert a string into Date
def ChangeDate(word : String) : Date = {
| return
TO_DATE(FROM_UNIXTIME(UNIX_TIMESTAMP(word),"dd/MM/"),"-MM-dd")
| }
:19: error: not found: type Date
That code to_date.. works OK in sql but not here. It is complaining about
to_da
