UDF in spark

Hi Everyone, I am new to spark.may I know how to define and use User Define Function in SPARK SQL. I want to use defined UDF by using sql queries. My Environment Windows 8 spark 1.3.1 Warm Regards, Vinod

Re: UDF in spark

com> wrote: > Hi, > > sqlContext.udf.register("udfname", functionname _) > > example: > > def square(x:Int):Int = { x * x} > > register udf as below > > sqlContext.udf.register("square",square _) > > Thanks, > Vishnu > > On Wed,

Re: UDF in spark

, VISHNU SUBRAMANIAN < johnfedrickena...@gmail.com> wrote: > HI Vinod, > > Yes If you want to use a scala or python function you need the block of > code. > > Only Hive UDF's are available permanently. > > Thanks, > Vishnu > > On Wed, Jul 8, 2015 at 5:17

Using Hive UDF in spark

Hi everyone Shall we use UDF defined in hive using spark sql? I've created a UDF in spark and registered it using sqlContext.udf.register,but when I restarted a service the UDF was not available. I've heared that Hive UDF's are permanently stored in hive.(Please Correct me if I am wrong). Thanks

Data Processing speed SQL Vs SPARK

Hi Everyone, I am new to spark. Am using SQL in my application to handle data in my application.I have a thought to move to spark now. Is data processing speed of spark better than SQL server? Thank, Vinod

Re: Data Processing speed SQL Vs SPARK

to process? > On 9 Jul 2015 22:28, "vinod kumar" wrote: > >> Hi Everyone, >> >> I am new to spark. >> >> Am using SQL in my application to handle data in my application.I have a >> thought to move to spark now. >> >> Is data processing speed of spark better than SQL server? >> >> Thank, >> Vinod >> >

Re: Data Processing speed SQL Vs SPARK

For records below 50,000 SQL is better right? On Fri, Jul 10, 2015 at 12:18 AM, ayan guha wrote: > With your load, either should be fine. > > I would suggest you to run couple of quick prototype. > > Best > Ayan > > On Fri, Jul 10, 2015 at 2:06 PM, vinod kumar >

SPARK vs SQL

Hi Everyone, Is there is any document/material which compares spark with SQL server? If so please share me the details. Thanks, Vinod

Caching in spark

Hi Guys, Can any one please share me how to use caching feature of spark via spark sql queries? -Vinod

Spark Intro

Hi Everyone, I am developing application which handles bulk of data around millions(This may vary as per user's requirement) records.As of now I am using MsSqlServer as back-end and it works fine but when I perform some operation on large data I am getting overflow exceptions.I heard about spark

Re: Spark Intro

15 at 3:50 AM, Akhil Das wrote: > This is where you can get started > https://spark.apache.org/docs/latest/sql-programming-guide.html > > Thanks > Best Regards > > On Mon, Jul 13, 2015 at 3:54 PM, vinod kumar > wrote: > >> >> Hi Everyone, >> >>

Re: Spark Intro

if >> you have more memory, also if you have enough cores 4 records are >> nothing. >> >> Thanks >> Best Regards >> >> On Tue, Jul 14, 2015 at 3:09 PM, vinod kumar >> wrote: >> >>> Hi Akhil >>> >>> Is my choice to switc

Cache in Spark

Hi Guys, May I know whether cache is enabled in spark by default? Thanks, Vinod

Re: Cache in Spark

s you tell it too. > > You have to explicitly call the cache method in your RDD: > linesWithSpark.cache() > > Thanks, > Natu > > > > > On Fri, Oct 9, 2015 at 10:47 AM, vinod kumar > wrote: > >> Hi Guys, >> >> May I know whether cache is enabled in spark by default? >> >> Thanks, >> Vinod >> > >

SQL Server to Spark

Hi Everyone, I am in need to use the table from MsSQLSERVER in SPARK.Any one please share me the optimized way for that? Thanks in advance, Vinod

Create table from local machine

Hi, I am in need to create a table in spark.for that I have uploaded a csv file in HDFS and created a table using following query CREATE EXTERNAL table IF NOT EXISTS " + tableName + " (teams string,runs int) " + "ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LOCATION '" + hdfspath + "'"; May I k

Functions in Spark SQL

Hi, May I know how to use the functions mentioned in http://spark.apache.org/docs/1.4.0/api/scala/index.html#org.apache.spark.sql.functions$ in spark sql? when I use like "Select last(column) from tablename" I am getting error like 15/07/27 03:00:00 INFO exec.FunctionRegistry: Unable to lookup

Re: Functions in Spark SQL

to paste your test code here ? And which version of Spark > are u using ? > > Best, > Sun. > > ------ > fightf...@163.com > > > *From:* vinod kumar > *Date:* 2015-07-27 15:04 > *To:* User > *Subject:* Functions in Spark SQ

Buffer Overflow exception

Hi, I am getting buffer over flow exception while using spark via thrifserver base.May I know how to overcome this? Code: HqlConnection con = new HqlConnection("localhost", 10001, HiveServer.HiveServer2); con.Open(); HqlCommand createCommand = new HqlCommand(tablequery,