foreachAsync at RemoteHiveSparkClient.java

hi,all i execute a sql on hive on spark,the comand like: select distinct st.sno,sname from student st join score sc on(st.sno=sc.sno) where sc.cno IN(11,12,13) and st.sage > 28;(some days ago this sql can work ) but it give me some Info in hive shell : Query Hive on Spark job[0] stages: 0 St

Hive UDFs?

Hi experts, I heard that if I want to use Hive UDFs, I must deploy the jars to all the machines that are running Hive (which is painful to me..) After reading this I didn't find any document talking about this - could someone help to

Re: Hive UDFs?

Hi Xiayong, Q 1 :- You need to deploy the jar in the server where your hive is running. Q 2 :- Which language you are looking instead of Java ? . As per my understanding you can write UDF functions in python also ( I have not tried this from my end ) Regards, Rathish On Thu, Dec 18, 2014 at 2:5

RE: Hive UDFs?

Thanks! For Q1 - is there a way that Hive helps me to automatically do this (for example, I can register the UDF somewhere and the UDF gets automatically distributed)? Or I need to login to each node to ensure this happens? Xiaoyong -Original Message- From: Rathish A M [mailto:rathis...

RE: Hive UDFs?

+ Rathish Xiaoyong -Original Message- From: Xiaoyong Zhu Sent: Thursday, December 18, 2014 5:56 PM To: user@hive.apache.org Subject: RE: Hive UDFs? Thanks! For Q1 - is there a way that Hive helps me to automatically do this (for example, I can register the UDF somewhere and the UDF get

Re: Row Delimiter in Create table

Thank you. On Wed, Dec 17, 2014 at 8:55 PM, Navis류승우 wrote: > > Afaik, it was restricted by implementation of hadoop. But now hadoop-2 > supports custom delimiter, hopefully it also can be implemented in hive. > > I'm not sure but currently possible way of do that is setting > "textinputformat.re

Re: Hive UDFs?

You only need the jar on the computer where you execute the hive command , not on all cluster nodes On Thu, Dec 18, 2014 at 2:55 AM, Xiaoyong Zhu wrote: > > Thanks! > For Q1 - is there a way that Hive helps me to automatically do this (for > example, I can register the UDF somewhere and the UDF

Re: Hive UDFs?

Another option is https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-PermanentFunctions, as another user mentioned on this list a few days ago. On Dec 18, 2014, at 5:54 AM, Stéphane Verlet wrote: > You only need the jar on the computer where you execute the h

Re: custom binary format

Hi thanks for the answer so far, however, I still think there must be an easy way. The file format I’m looking at is pretty simple. There is first an header of n bytes, Which can be ignored. After that there is the data. The data consists of rows where ich rows has 9 bytes. First there is a byt

Re: custom binary format

Hi Ingo, Take a look at https://hadoop.apache.org/docs/r2.3.0/api/org/apache/hadoop/mapred/FixedLengthInputFormat.html--it seems to be designed for use cases very similar to yours. You may need to subclass it to make things work precisely the way you need (in particular, to deal with the head

Re: custom binary format

Hello Andrew, this one looks indeed like a good idea. However, there is also another Problem already here. This InputFormat expects that conf.setInt(FixedLengthInputFormat.FIXED_RECORD_LENGTH, recordLength); is set. I haven’t found any way to specify a parameter for a InputFormat. I couldn’t fin

Re: custom binary format

So in hive you can actually do that via the SET command (documented here https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Cli) as follows: hive> SET fixedlengthinputformat.record.length = This value will be passed through to the JobConf, and the input format ought to pick it u

RE: Hive UDFs?

I see, thanks! Xiaoyong From: Jason Dere [mailto:jd...@hortonworks.com] Sent: Friday, December 19, 2014 3:52 AM To: user@hive.apache.org Subject: Re: Hive UDFs? Another option is https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-PermanentFunctions, as anothe

RE: relationship between a hive query and a tez dag

Sorry to update this again - but why don't we do a cross query optimization and make the query into 1 DAG (if all the queries in a certain script are linked with each other).. this seems a more optimized way.. Xiaoyong From: Xiaoyong Zhu [mailto:xiaoy...@microsoft.com] Sent: Thursday, December