Re: Hive to HDFS directly using INSERT OVERWRITE DIRECTORY Imcompatible issue

You could create an external table at a location of your choice with the format desired. Then do a select into the table. The data at the location of your table will be in the format you desire, which you can copy over. Best Regards, Sonal Nube Technologies

RCFile with hive.optimize.cp setting

In our environment, our data are put on Amazon S3, and data are in RCFile format. In order to make Hive queries work, we found that we have to change the hive.optimize.cp to false. Otherwise, some queries will fail. Now, when we try some complicated queries with multiple subqueries and joins, we

Fwd: is that possible to use Hive 0.11 with libthrift 0.7

Hi Johnny, Sorry for the delayed response (was offline for a few days). So the code that actually uses Thrift in is the Hive core JAR, which is what will get shaded. The easiest thing to do is add this target to the root build.xml file: Notice this assumes the JAR file f

Hive to HDFS directly using INSERT OVERWRITE DIRECTORY Imcompatible issue

Hi: Currently, our hive_to_hdfs function has two parts. The first part retrieves transactions records in Hive, put into a temporary file in local file system.The second part puts temporary file in local file system into HDFS. The second part work on NameNode and is outside of Hadoop process and

Re: Dynamic Partitioning not working in HCatalog 0.11?

Wanted to follow-up with the solution to this issue I raised last week. I upgraded to Hive 0.12 branch and added these two properties to my Pig script: SET hive.exec.dynamic.partition.mode nonstrict; SET hive.exec.dynamic.partition true; This seems to do the trick although I don't recall having t

Re: Use hive on hadoop 3.0.0

If you are using HIVE-TEZ branch, you also need tez (git clone g...@github.com:apache/tez.git). Publish the tez artifacts locally (mvn clean install -DskipTests). In hive, change build.properties to have hadoop-0.23.version=Hadoop-3.0.0-SNAPSHOT. Specifying that using "-Dhadoop-0.23.version" in yo

Re: Custom SerDe: Initialize() passes a null configuration to my Custom SerDe

Thank you all for the lightning fast replies. @Yin: I am actually passing SerDe properties and they are working fine. Actually, this is how I pass the the full namespace of the class I want to use. I receive the Properties object fine, it's the Configuration one that is null. :( @Hari: Perfect Ha

[RESULT] Apache Hive 0.12.0 Release Candidate 1

With 3 binding +1's and 2 non-binding +1s and no -1s this vote passes. Thanks to everybody who voted and gave feedback. I will start working on publishing the release. Thanks, Thejas On Sun, Oct 13, 2013 at 4:37 PM, Carl Steinbach wrote: > +1 (binding) > > >> Regarding the 3 day deadline for v

Re: [VOTE] Apache Hive 0.12.0 Release Candidate 1

Poking more it looks like this affects not just webhcatalog scripts but just all hcatalog scripts. Consequently, it looks like starting hcatalog server wouldn't work either. I will stop spamming multiple mailing lists now and post more updates, if any, on the JIRA (HIVE-5534) itself. On Mon, Oct

Re: [VOTE] Apache Hive 0.12.0 Release Candidate 1

I wasn't able to start webhcat server. It seems to be related to file permissions. FWIW, it's not a regression (the same problem existed in Hive 0.11). Having said that, it makes webhcat pretty unusable out of the box. I created HIVE-5534 to track this. On Sun, Oct 13, 2013 at 4:37 PM, Carl Stei

Re: Custom SerDe: Initialize() passes a null configuration to my Custom SerDe

Have you seen? https://github.com/edwardcapriolo/hive-protobuf/ On Mon, Oct 14, 2013 at 8:16 AM, Rui Martins wrote: > Hi hive users, > > I am writing a custom SerDe that loads any protocol buffer generated > class. > For flexibility this class can live in a jar external to the SerDe's jar > an

Trouble building Hive 0.12 branch against Hadoop 2.0.0-cdh4.1.2

Checked out 0.12 branch and am getting the following error when building with this command: ant clean package -Dhadoop.version=2.0.0-cdh4.1.2 -Dhadoop-0.23.version=2.0.0-cdh4.1.2 -Dhadoop.mr.rev=23 -Dmvn.hadoop.profile=hadoop23 ERROR: [mkdir] Created dir: /data-volume/home/pipeline/dev/hive

ClassCastExceptions using Hiveserver2 Thrift interface, Hive 0.11

Hi there, We're using the rbhive gem (https://github.com/forward3d/rbhive) to connect to the Hiveserver2 Thrift service running on Hive 0.11. I've turned SASL auth off with the following in hive-site.xml: hive.server2.authentication NOSASL When I connect, I get the following stack trace i

Re: Custom SerDe: Initialize() passes a null configuration to my Custom SerDe

Please see https://svn.apache.org/repos/asf/hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java https://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/ to see how a non-null conf is passed to initialize() Thanks Hari On Mon, Oct 14, 2013 at 6:29 PM

Re: Custom SerDe: Initialize() passes a null configuration to my Custom SerDe

Can you try to set serde properties? https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-AddSerDeProperties I have not tried it, but seems it is the right way to pass configurations to serde class. Thanks, Yin On Mon, Oct 14, 2013 at 8:20 AM, Rui Martins wrot

Re: Custom SerDe: Initialize() passes a null configuration to my Custom SerDe

+dev hive mailing list that I should've mailed in the first place. (apologies for the spam) On Mon, Oct 14, 2013 at 1:16 PM, Rui Martins wrote: > Hi hive users, > > I am writing a custom SerDe that loads any protocol buffer generated > class. > For flexibility this class can live in a jar exte

Custom SerDe: Initialize() passes a null configuration to my Custom SerDe

Hi hive users, I am writing a custom SerDe that loads any protocol buffer generated class. For flexibility this class can live in a jar external to the SerDe's jar and then I just use the Hive Configuration class passed in the initiliaze to dynamically load it and set the schema for the Hive table