Re: 回复： different outer join plan between hive 0.9 and hive 0.10

In short, hive tags rows a filter mask as a short for outer join, which can contain 16 flags. (see HIVE-3411, plz) I'll survey for a solution. 2013/5/14 wzc1989 : > "hive cannot merge joins of 16+ aliases with outer join into single stage." > In our use case we use one table full outer join all o

回复： different outer join plan between hive 0.9 and hive 0.10

"hive cannot merge joins of 16+ aliases with outer join into single stage." In our use case we use one table full outer join all other table to produce one big table, which may exceed 16 outer join limits and will be split into multi stage under hive 0.10. It become very slow under hive 0.10 whil

Re: 回复： different outer join plan between hive 0.9 and hive 0.10

The error message means hive cannot merge joins of 16+ aliases with outer join into single stage. It was 8 way originally (HIVE-3411) but expanded to 16 later. Check https://issues.apache.org/jira/browse/HIVE-3411 for details. 2013/5/14 wzc1989 : > This time i cherry-pick HIVE-3464, HIVE-4212, H

adding "create or replace view" documentation to the wiki

I just realized that this has existed in Hive for a couple years now, but the only reference to it that exists on this page in the wiki https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL is an offhand remark in the documentation for ALTER VIEW AS SELECT. Looks like it was implement

Re: HADOOP and Query Capabilities

On Mon, May 13, 2013 at 9:34 AM, Nalin Khosla wrote: > Had a quick question wrt to querying HADOOP data; > > 1. What tools are available to Query Hadoop data in real time vs batch? > The line between real time and batch isn't that clear. We are working on substantially speeding up the performance

Re: select failed in Hive. java.io.FileNotFoundException

Hi Gautier, Looks you're running Hive on top of local filesystem. The exception stack said you haven't created staging directory. You can try creating the directory manually, or changing the config parameter "mapreduce.jobtracker.staging.root.dir" to a valid directory. On Mon, May 13, 2013 at 4:

Re: HADOOP and Query Capabilities

Sorry for the typo in the 3rd answer. I meant, question 1 covers this. Warm Regards, Tariq cloudfront.blogspot.com On Mon, May 13, 2013 at 10:13 PM, Mohammad Tariq wrote: > Hello Nalin, > > Please find my comments embedded below : > > 1. What tools are available to Query Hadoop data in re

Re: Trying to write a custom HiveOutputFormat

You could also look at the OrcSerde and how it works. https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSerde.java Basically, OrcSerde on "serialize" just wraps the row and object inspector in a fake writable. That is passed down to the OutputFormat. On "de

Re: HADOOP and Query Capabilities

Hello Nalin, Please find my comments embedded below : 1. What tools are available to Query Hadoop data in real time vs batch? >> Real time : Cloudera Impala, Batch : Hive and Pig..You might wanna have a look at Apache Drill as well. 2. I bel

Re: Trying to write a custom HiveOutputFormat

Cool, thanks for the clarification guys. I'll get on with the implementation of another SerDe. :) @Edward - Thanks for the link. I actually already have a protobuffer SerDe that I implemented that allows passing any protobuffer schema into it (as long as it's proto 2.4.1) and also has some other f

HADOOP and Query Capabilities

Had a quick question wrt to querying HADOOP data; 1. What tools are available to Query Hadoop data in real time vs batch? 2. I believe HIVE provides a batch interface, not sure on what tools within HIVE support the query capabilities against HADOOP ? 3. Besides HIVE, are there any other Query

回复： different outer join plan between hive 0.9 and hive 0.10

This time i cherry-pick HIVE-3464, HIVE-4212, HIVE-4206 and some related commits and the above explain result matches in hive 0.9 and hive 0.10, thanks! But I confuse about this error msg: JOINNODE_OUTERJOIN_MORETHAN_16(10142, "Single join node containing outer join(s) " + "cannot have mo

Re: Trying to write a custom HiveOutputFormat

You need to use a combination of output format and serde, this might allow you to do something like present struct objects to the input format rather then Text objects. You may want to take a look at the protobuf input format we use: https://github.com/edwardcapriolo/hive-protobuf/ You could reve

Re: Trying to write a custom HiveOutputFormat

You may need to implement a custom SerDe. Regards, Ramki On May 13, 2013, at 5:14 AM, Rui Martins wrote: > Hi guys, > > I'm currently writing my on HiveOutputFormat as I would like to write the > output of hive queries into a specific protobuf format my team is using. > I have managed to do

Re: Who is the hive admin user?

Thanks a lot Owen & Lefty. Sorry for the late reply, got a bit busy during the weekend. So... what I understand is: 1. To enable security in Hive you need minimum of Hive 0.10 and Kerberos enabled. 2. The best option is to protect the HDFS directories that the data is stored in. 3. The user roles

Trying to write a custom HiveOutputFormat

Hi guys, I'm currently writing my on HiveOutputFormat as I would like to write the output of hive queries into a specific protobuf format my team is using. I have managed to do this however, the Writable object I get from Hive as a result of a SELECT query is of type Text. This means that I have t

Re: select failed in Hive. java.io.FileNotFoundException

Thank's for your response. I'm really a beginner, so I don't really understand what you mean by : "correctly" 2013/5/13 shashwat shriparv > Set the hive class path correctly. > > *Thanks & Regards* > > ∞ > Shashwat Shriparv > > > > On Mon, May 13, 2013 at 1:10 PM, gautier vivard wrote: > >>

Re: select failed in Hive. java.io.FileNotFoundException

Set the hive class path correctly. *Thanks & Regards* ∞ Shashwat Shriparv On Mon, May 13, 2013 at 1:10 PM, gautier vivard wrote: > Hey, > > I'm beginner in the hadoop world, and more specifically in Hive (I work on > hive-0.9.0 on Ubuntu). I work on MapR M3 in standalone mode. I'm trainin

select failed in Hive. java.io.FileNotFoundException

Hey, I'm beginner in the hadoop world, and more specifically in Hive (I work on hive-0.9.0 on Ubuntu). I work on MapR M3 in standalone mode. I'm training on hive and have some issues with one query : > SELECT year FROM records; Automatically selecting local only mode for query Total MapReduce job