Re: Trying to write a custom HiveOutputFormat

2013-05-13 Thread Owen O'Malley
You could also look at the OrcSerde and how it works. https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSerde.java Basically, OrcSerde on "serialize" just wraps the row and object inspector in a fake writable. That is passed down to the OutputFormat. On "de

Re: Trying to write a custom HiveOutputFormat

2013-05-13 Thread Rui Martins
Cool, thanks for the clarification guys. I'll get on with the implementation of another SerDe. :) @Edward - Thanks for the link. I actually already have a protobuffer SerDe that I implemented that allows passing any protobuffer schema into it (as long as it's proto 2.4.1) and also has some other f

Re: Trying to write a custom HiveOutputFormat

2013-05-13 Thread Edward Capriolo
You need to use a combination of output format and serde, this might allow you to do something like present struct objects to the input format rather then Text objects. You may want to take a look at the protobuf input format we use: https://github.com/edwardcapriolo/hive-protobuf/ You could reve

Re: Trying to write a custom HiveOutputFormat

2013-05-13 Thread Ramki Gmail
You may need to implement a custom SerDe. Regards, Ramki On May 13, 2013, at 5:14 AM, Rui Martins wrote: > Hi guys, > > I'm currently writing my on HiveOutputFormat as I would like to write the > output of hive queries into a specific protobuf format my team is using. > I have managed to do