[ 
https://issues.apache.org/jira/browse/HIVE-3874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13583385#comment-13583385
 ] 

Phabricator commented on HIVE-3874:
-----------------------------------

omalley has commented on the revision "HIVE-3874 [jira] Create a new Optimized 
Row Columnar file format for Hive".

  | Do you want to have a simple test for HIVE-4015 as part of this patch ?

  Since the test will fail until I change the grammer, I think it would be 
better to wait until they will pass.

  | 1. Can you add more comments - specially, in the class/interface 
definitions Writer/TreeWriter/StreamFactory to name a few.

  Sure.

  | 2. Can column statistics be made optional ? (can be a follow-up)

  They are very cheap in practice, but it wouldn't be hard to disable them.

  | 3. This has a lot of new code - I mean, is it possible to use some of the 
constructs which are already there - for eg. RedBlackTrees, RLE etc. Can you 
use some existing implementations instead of writing these from scratch ?

  I'm a big fan of not writing new code when I can just use someone else's. 
That said, it is always a trade off when evaluating when to re-use versus write 
new that involves comparing the requirements to what the other code provides.

  I'm not aware of any open source java red-black trees that work on primitives 
without allocating multiple objects per an entry. Do you have a suggestion?

  The RLE is very specific to ORC format and it didn't seem like any of the 
implementations available were a good match. I'm also considering how to do a 
better delta and small integer encoding, but I'll do that in a follow up jira.

  | Right now, the RLE is fixed. Should it be pluggable ? I mean - we can have 
a different scheme to store deltas.

  I think that pluggable will create compatibility problems, since you won't be 
able to read an ORC file that was written by a different plugin.

  My preferred direction is to use the ColumnEncoding to allow the Writer to 
pick a different encoding based on the observed data. For example, by looking 
at the first 100,000 values the writer should be able to decide if a dictionary 
or direct encoding is better. We could use the same mechanism to add additional 
encodings.

INLINE COMMENTS
  ql/src/java/org/apache/hadoop/hive/ql/orc/OutStream.java:136-140 There is an 
requirement that the codec's compress method will return false rather than take 
more space than the input. Given that, if the compressed is empty, we can't 
need the overflow.
  ql/src/java/org/apache/hadoop/hive/ql/orc/OrcInputFormat.java:149-151 I've 
removed it.
  ql/src/java/org/apache/hadoop/hive/ql/orc/OrcStruct.java:307 fixed.
  ql/src/java/org/apache/hadoop/hive/ql/orc/WriterImpl.java:561-562 I've added 
the size of the dictionary to the estimate of the memory size, which should be 
better.
  ql/src/java/org/apache/hadoop/hive/ql/orc/BitFieldReader.java:18 i managed to 
move the directory in the wrong place. fixed.

REVISION DETAIL
  https://reviews.facebook.net/D8529

To: JIRA, omalley
Cc: kevinwilfong, njain

                
> Create a new Optimized Row Columnar file format for Hive
> --------------------------------------------------------
>
>                 Key: HIVE-3874
>                 URL: https://issues.apache.org/jira/browse/HIVE-3874
>             Project: Hive
>          Issue Type: Improvement
>          Components: Serializers/Deserializers
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: hive.3874.2.patch, HIVE-3874.D8529.1.patch, 
> HIVE-3874.D8529.2.patch, HIVE-3874.D8529.3.patch, OrcFileIntro.pptx, orc.tgz
>
>
> There are several limitations of the current RC File format that I'd like to 
> address by creating a new format:
> * each column value is stored as a binary blob, which means:
> ** the entire column value must be read, decompressed, and deserialized
> ** the file format can't use smarter type-specific compression
> ** push down filters can't be evaluated
> * the start of each row group needs to be found by scanning
> * user metadata can only be added to the file when the file is created
> * the file doesn't store the number of rows per a file or row group
> * there is no mechanism for seeking to a particular row number, which is 
> required for external indexes.
> * there is no mechanism for storing light weight indexes within the file to 
> enable push-down filters to skip entire row groups.
> * the type of the rows aren't stored in the file

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Reply via email to