[
https://issues.apache.org/jira/browse/HIVE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-5199:
----------------------------------------------------
Description:
The issue happens because of the changes in HIVE-3833.
Consider a partitioned table with different custom serdes for the partition and
tables. The serde at table level, say, customSerDe1's object inspector is of
settableDataType where as the serde at partition level, say, customSerDe2's
object inspector is of nonSettableDataType. The current implementation
introduced by HIVE-3833 does not convert nested Complex Data Types which extend
nonSettableObjectInspector to a settableObjectInspector type inside
ObjectInspectorConverters.getConvertedOI(). However, it tries to typecast the
nonSettableObjectInspector to a settableObjectInspector inside
ObjectInspectorConverters.getConverter(ObjectInspector inputOI, ObjectInspector
outputOI).
The attached patch HIVE-5199.2.patch.txt contains a stand-alone test case.
The below exception can happen via FetchOperator as well as MapOperator.
For example, consider the FetchOperator.
Inside FetchOperator consider the following call:
getRecordReader()->ObjectInspectorConverters. getConverter()
The output object inspector is of settable type(because it is generated via
ObjectInspectorConverters.getConvertedOI()) where as the input object inspector
that gets passed as serde.getObjectorInspector() and is non-settable. Inside
getConverter(), the (inputOI.equals(outputOI)) check fails and the switch
statement tries to cast the non-settable object inspector to a settable object
inspector.
The stack trace as follows:
2013-08-28 17:57:25,307 ERROR CliDriver (SessionState.java:printError(432)) -
Failed with exception java.io.IOException:java.lang.ClassCastException:
com.skype.data.whaleshark.hadoop.hive.proto.ProtoMapObjectInspector cannot be
cast to org.apache.hadoop.hive.serde2.objectinspector.SettableMapObjectInspector
java.io.IOException: java.lang.ClassCastException:
com.skype.data.whaleshark.hadoop.hive.proto.ProtoMapObjectInspector cannot be
cast to org.apache.hadoop.hive.serde2.objectinspector.SettableMapObjectInspector
at
org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:544)
at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:488)
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:136)
at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1412)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:271)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:756)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
Caused by: java.lang.ClassCastException:
com.skype.data.whaleshark.hadoop.hive.proto.ProtoMapObjectInspector cannot be
cast to org.apache.hadoop.hive.serde2.objectinspector.SettableMapObjectInspector
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConverter(ObjectInspectorConverters.java:144)
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters$StructConverter.<init>(ObjectInspectorConverters.java:307)
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConverter(ObjectInspectorConverters.java:138)
at
org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:406)
at
org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:508)
was:
Custom serdes which used to work in HDP 1.1 is no longer working with HDP 1.3.
The issue happens when the partition serde is not of settable type in HDP 1.3.
The below exception happens via FetchOperator as well as MapOperator.
Inside FetchOperator consider the following call:
getRecordReader()->ObjectInspectorConverters. getConverter()
The output object inspector is of settable type(because it is generated via
ObjectInspectorConverters.getConvertedOI()) where as the input object inspector
that gets passed as serde.getObjectorInspector() and is non-settable. Inside
getConverter(), the (inputOI.equals(outputOI)) check fails and the switch
statement tries to cast the non-settable object inspector to a settable object
inspector.
The stack trace as follows:
2013-08-28 17:57:25,307 ERROR CliDriver (SessionState.java:printError(432)) -
Failed with exception java.io.IOException:java.lang.ClassCastException:
com.skype.data.whaleshark.hadoop.hive.proto.ProtoMapObjectInspector cannot be
cast to org.apache.hadoop.hive.serde2.objectinspector.SettableMapObjectInspector
java.io.IOException: java.lang.ClassCastException:
com.skype.data.whaleshark.hadoop.hive.proto.ProtoMapObjectInspector cannot be
cast to org.apache.hadoop.hive.serde2.objectinspector.SettableMapObjectInspector
at
org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:544)
at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:488)
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:136)
at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1412)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:271)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:756)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
Caused by: java.lang.ClassCastException:
com.skype.data.whaleshark.hadoop.hive.proto.ProtoMapObjectInspector cannot be
cast to org.apache.hadoop.hive.serde2.objectinspector.SettableMapObjectInspector
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConverter(ObjectInspectorConverters.java:144)
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters$StructConverter.<init>(ObjectInspectorConverters.java:307)
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConverter(ObjectInspectorConverters.java:138)
at
org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:406)
at
org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:508)
Affects Version/s: 0.11.0
Summary: Custom SerDe containing a nonSettable complex data type
row object inspector throws cast exception with HIVE 0.11 (was: Read Only
Custom SerDe works with HDP 1.1 but not with HDP 1.3)
> Custom SerDe containing a nonSettable complex data type row object inspector
> throws cast exception with HIVE 0.11
> -----------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-5199
> URL: https://issues.apache.org/jira/browse/HIVE-5199
> Project: Hive
> Issue Type: Bug
> Affects Versions: 0.11.0
> Reporter: Hari Sankar Sivarama Subramaniyan
> Assignee: Hari Sankar Sivarama Subramaniyan
> Priority: Critical
> Attachments: HIVE-5199.patch.txt
>
>
> The issue happens because of the changes in HIVE-3833.
> Consider a partitioned table with different custom serdes for the partition
> and tables. The serde at table level, say, customSerDe1's object inspector is
> of settableDataType where as the serde at partition level, say,
> customSerDe2's object inspector is of nonSettableDataType. The current
> implementation introduced by HIVE-3833 does not convert nested Complex Data
> Types which extend nonSettableObjectInspector to a settableObjectInspector
> type inside ObjectInspectorConverters.getConvertedOI(). However, it tries to
> typecast the nonSettableObjectInspector to a settableObjectInspector inside
> ObjectInspectorConverters.getConverter(ObjectInspector inputOI,
> ObjectInspector outputOI).
> The attached patch HIVE-5199.2.patch.txt contains a stand-alone test case.
> The below exception can happen via FetchOperator as well as MapOperator.
> For example, consider the FetchOperator.
> Inside FetchOperator consider the following call:
> getRecordReader()->ObjectInspectorConverters. getConverter()
> The output object inspector is of settable type(because it is generated via
> ObjectInspectorConverters.getConvertedOI()) where as the input object
> inspector that gets passed as serde.getObjectorInspector() and is
> non-settable. Inside getConverter(), the (inputOI.equals(outputOI)) check
> fails and the switch statement tries to cast the non-settable object
> inspector to a settable object inspector.
> The stack trace as follows:
> 2013-08-28 17:57:25,307 ERROR CliDriver (SessionState.java:printError(432)) -
> Failed with exception java.io.IOException:java.lang.ClassCastException:
> com.skype.data.whaleshark.hadoop.hive.proto.ProtoMapObjectInspector cannot be
> cast to
> org.apache.hadoop.hive.serde2.objectinspector.SettableMapObjectInspector
> java.io.IOException: java.lang.ClassCastException:
> com.skype.data.whaleshark.hadoop.hive.proto.ProtoMapObjectInspector cannot be
> cast to
> org.apache.hadoop.hive.serde2.objectinspector.SettableMapObjectInspector
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:544)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:488)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:136)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1412)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:271)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:756)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
> Caused by: java.lang.ClassCastException:
> com.skype.data.whaleshark.hadoop.hive.proto.ProtoMapObjectInspector cannot be
> cast to
> org.apache.hadoop.hive.serde2.objectinspector.SettableMapObjectInspector
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConverter(ObjectInspectorConverters.java:144)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters$StructConverter.<init>(ObjectInspectorConverters.java:307)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConverter(ObjectInspectorConverters.java:138)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:406)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:508)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
