[ 
https://issues.apache.org/jira/browse/HIVE-6711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth J updated HIVE-6711:
-----------------------------

    Description: 
HIVE-6707 had issues with map size. getMapSize() of LazyMap and LazyBinaryMap 
does not deserialize the keys and count the number of unique keys. Since 
getMapSize() may return non-distinct count of keys, the length of maps stored 
using ORC's map tree writer will not be in sync with actual map size. As a 
result of this RLE reader will try to read beyond the disk range expecting more 
map entries and will throw exception.

Stack trace will look like:
{code}
Caused by: java.io.EOFException: Read past end of RLE integer from compressed 
stream Stream for column 2 kind DATA position: 22059699 length: 22059699 range: 
0 offset: 22359014 limit: 22359014 range 0 = 0 to 22059699 uncompressed: 53370 
to 53370
        at 
org.apache.hadoop.hive.ql.io.orc.RunLengthIntegerReaderV2.readValues(RunLengthIntegerReaderV2.java:54)
        at 
org.apache.hadoop.hive.ql.io.orc.RunLengthIntegerReaderV2.next(RunLengthIntegerReaderV2.java:301)
        at 
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$StringDictionaryTreeReader.next(RecordReaderImpl.java:1572)
        at 
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$StringTreeReader.next(RecordReaderImpl.java:1330)
        at 
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$MapTreeReader.next(RecordReaderImpl.java:2041)
        at 
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$StructTreeReader.next(RecordReaderImpl.java:1772)
        at 
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.next(RecordReaderImpl.java:2963)
        at 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.next(OrcInputFormat.java:121)
{code}

  was:HIVE-6707 had issues with map size. getMapSize() of LazyMap and 
LazyBinaryMap does not deserialize the keys and count the number of unique 
keys. Since getMapSize() may return non-distinct count of keys, the length of 
maps stored using ORC's map tree writer will not be in sync with actual map 
size. As a result of this RLE reader will try to read beyond the disk range 
expecting more map entries and will throw exception.


> ORC maps uses getMapSize() from MapOI which is unreliable
> ---------------------------------------------------------
>
>                 Key: HIVE-6711
>                 URL: https://issues.apache.org/jira/browse/HIVE-6711
>             Project: Hive
>          Issue Type: Bug
>    Affects Versions: 0.11.0, 0.12.0, 0.13.0
>            Reporter: Prasanth J
>            Assignee: Prasanth J
>              Labels: orcfile
>             Fix For: 0.13.0, 0.14.0
>
>         Attachments: HIVE-6711.1.patch
>
>
> HIVE-6707 had issues with map size. getMapSize() of LazyMap and LazyBinaryMap 
> does not deserialize the keys and count the number of unique keys. Since 
> getMapSize() may return non-distinct count of keys, the length of maps stored 
> using ORC's map tree writer will not be in sync with actual map size. As a 
> result of this RLE reader will try to read beyond the disk range expecting 
> more map entries and will throw exception.
> Stack trace will look like:
> {code}
> Caused by: java.io.EOFException: Read past end of RLE integer from compressed 
> stream Stream for column 2 kind DATA position: 22059699 length: 22059699 
> range: 0 offset: 22359014 limit: 22359014 range 0 = 0 to 22059699 
> uncompressed: 53370 to 53370
>         at 
> org.apache.hadoop.hive.ql.io.orc.RunLengthIntegerReaderV2.readValues(RunLengthIntegerReaderV2.java:54)
>         at 
> org.apache.hadoop.hive.ql.io.orc.RunLengthIntegerReaderV2.next(RunLengthIntegerReaderV2.java:301)
>         at 
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$StringDictionaryTreeReader.next(RecordReaderImpl.java:1572)
>         at 
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$StringTreeReader.next(RecordReaderImpl.java:1330)
>         at 
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$MapTreeReader.next(RecordReaderImpl.java:2041)
>         at 
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$StructTreeReader.next(RecordReaderImpl.java:1772)
>         at 
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.next(RecordReaderImpl.java:2963)
>         at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.next(OrcInputFormat.java:121)
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Reply via email to