[ https://issues.apache.org/jira/browse/HIVE-3874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13583385#comment-13583385 ]
Phabricator commented on HIVE-3874: ----------------------------------- omalley has commented on the revision "HIVE-3874 [jira] Create a new Optimized Row Columnar file format for Hive". | Do you want to have a simple test for HIVE-4015 as part of this patch ? Since the test will fail until I change the grammer, I think it would be better to wait until they will pass. | 1. Can you add more comments - specially, in the class/interface definitions Writer/TreeWriter/StreamFactory to name a few. Sure. | 2. Can column statistics be made optional ? (can be a follow-up) They are very cheap in practice, but it wouldn't be hard to disable them. | 3. This has a lot of new code - I mean, is it possible to use some of the constructs which are already there - for eg. RedBlackTrees, RLE etc. Can you use some existing implementations instead of writing these from scratch ? I'm a big fan of not writing new code when I can just use someone else's. That said, it is always a trade off when evaluating when to re-use versus write new that involves comparing the requirements to what the other code provides. I'm not aware of any open source java red-black trees that work on primitives without allocating multiple objects per an entry. Do you have a suggestion? The RLE is very specific to ORC format and it didn't seem like any of the implementations available were a good match. I'm also considering how to do a better delta and small integer encoding, but I'll do that in a follow up jira. | Right now, the RLE is fixed. Should it be pluggable ? I mean - we can have a different scheme to store deltas. I think that pluggable will create compatibility problems, since you won't be able to read an ORC file that was written by a different plugin. My preferred direction is to use the ColumnEncoding to allow the Writer to pick a different encoding based on the observed data. For example, by looking at the first 100,000 values the writer should be able to decide if a dictionary or direct encoding is better. We could use the same mechanism to add additional encodings. INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/orc/OutStream.java:136-140 There is an requirement that the codec's compress method will return false rather than take more space than the input. Given that, if the compressed is empty, we can't need the overflow. ql/src/java/org/apache/hadoop/hive/ql/orc/OrcInputFormat.java:149-151 I've removed it. ql/src/java/org/apache/hadoop/hive/ql/orc/OrcStruct.java:307 fixed. ql/src/java/org/apache/hadoop/hive/ql/orc/WriterImpl.java:561-562 I've added the size of the dictionary to the estimate of the memory size, which should be better. ql/src/java/org/apache/hadoop/hive/ql/orc/BitFieldReader.java:18 i managed to move the directory in the wrong place. fixed. REVISION DETAIL https://reviews.facebook.net/D8529 To: JIRA, omalley Cc: kevinwilfong, njain > Create a new Optimized Row Columnar file format for Hive > -------------------------------------------------------- > > Key: HIVE-3874 > URL: https://issues.apache.org/jira/browse/HIVE-3874 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers > Reporter: Owen O'Malley > Assignee: Owen O'Malley > Attachments: hive.3874.2.patch, HIVE-3874.D8529.1.patch, > HIVE-3874.D8529.2.patch, HIVE-3874.D8529.3.patch, OrcFileIntro.pptx, orc.tgz > > > There are several limitations of the current RC File format that I'd like to > address by creating a new format: > * each column value is stored as a binary blob, which means: > ** the entire column value must be read, decompressed, and deserialized > ** the file format can't use smarter type-specific compression > ** push down filters can't be evaluated > * the start of each row group needs to be found by scanning > * user metadata can only be added to the file when the file is created > * the file doesn't store the number of rows per a file or row group > * there is no mechanism for seeking to a particular row number, which is > required for external indexes. > * there is no mechanism for storing light weight indexes within the file to > enable push-down filters to skip entire row groups. > * the type of the rows aren't stored in the file -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira