[
https://issues.apache.org/jira/browse/HIVE-19494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16471216#comment-16471216
]
Prasanth Jayachandran commented on HIVE-19494:
----------------------------------------------
Only streaming ingest encountered this recently, but technically this can
happen for the case where multiple hive versions are involved. Spark compiles
against 1.2.1 but streaming ingest API requires hive 3.0.0 and in its codepath,
so only this instantiation in hive streaming API fails as of now (in my limited
testing). I left the config generic enough for future in case if someone runs
into the same issue they can reuse the hive config.
> Accept shade prefix during reflective instantiation of output format
> --------------------------------------------------------------------
>
> Key: HIVE-19494
> URL: https://issues.apache.org/jira/browse/HIVE-19494
> Project: Hive
> Issue Type: Sub-task
> Components: Streaming
> Affects Versions: 3.0.0, 3.1.0
> Reporter: Prasanth Jayachandran
> Assignee: Prasanth Jayachandran
> Priority: Major
> Attachments: HIVE-19494.1.patch
>
>
> Hive Streaming API jars are sometimes shaded with a different prefix when
> used in environments where another version of hive already exists (spark for
> example). In most cases, shading is done with rename of classes with some
> prefix. If an uber/assembly jar is generated with renamed prefix, Hive
> Streaming API will not work as Hive Streaming API will reflectively
> instantiate outputformat class using FQCN string provided by metastore table
> storage descriptor object.
> For example:
> RecordWriter will create instance of OutputFormat using string
> "org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat". When a shaded jar with
> renamed class references are used, this class will not be found by the
> classloader.
> We can optionally accept a shade prefix from user via config which will be
> tried (as fallback) when ClassNotFoundException is thrown.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
