RE: Accessing Table Properies from InputFormat

2013-05-29 Thread Peter Marron
me to examine the table properties and the predicate and to (pre-)filter the records returned? Or are you saying that what I am trying to do doesn't make sense? Z From: Edward Capriolo [mailto:edlinuxg...@gmail.com] Sent: 28 May 2013 16:45 To: user@hive.apache.org Cc: Peter Marron Sub

Re: Accessing Table Properies from InputFormat

2013-05-28 Thread Edward Capriolo
ORC, Parquet, and "new ones" are ... new. They do not constitute a huge portion of the user base if they constitute any at all. I do see a case for what you are describing, currently there are input formats that do properties via the configuration to the task. Also I feel like some of the confusio

Re: Accessing Table Properies from InputFormat

2013-05-28 Thread Owen O'Malley
On Tue, May 28, 2013 at 9:27 AM, Edward Capriolo wrote: > The question we are diving into is how much of hive is going to be > designed around edge cases? Hive really was not made for columnar formats, > or self describing data-types. For the most part it handles them fairly > well. > I don't vie

Re: Accessing Table Properies from InputFormat

2013-05-28 Thread Edward Capriolo
Right, hive discarding the key is rather annoying. I have a series of key+value input formats key only input format etc. Having hive return both the key and the value, would be a breaking change, but not be very difficult. The question we are diving into is how much of hive is going to be designed

Re: Accessing Table Properies from InputFormat

2013-05-28 Thread Owen O'Malley
On Tue, May 28, 2013 at 8:45 AM, Edward Capriolo wrote: > That does not really make sense. Your breaking the layered approache. > InputFormats read/write data, serdes interpret data based on the table > definition. its like asking "Why can't my input format run assembly code?" > The current model

Re: Accessing Table Properies from InputFormat

2013-05-28 Thread Edward Capriolo
That does not really make sense. Your breaking the layered approache. InputFormats read/write data, serdes interpret data based on the table definition. its like asking "Why can't my input format run assembly code?" On Tue, May 28, 2013 at 11:42 AM, Owen O'Malley wrote: > > > > On Tue, May 28,

Re: Accessing Table Properies from InputFormat

2013-05-28 Thread Owen O'Malley
On Tue, May 28, 2013 at 7:59 AM, Peter Marron < peter.mar...@trilliumsoftware.com> wrote: > Hi, > > ** ** > > Hive 0.10.0 over Hadoop 1.0.4. > > ** ** > > Further to my filtering questions of before. > > I would like to be able to access the table properties from inside my > custom In