[
https://issues.apache.org/jira/browse/HIVE-18410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-18410:
------------------------------------
Resolution: Fixed
Fix Version/s: (was: 2.3.2)
Status: Resolved (was: Patch Available)
Pushed to master. Thanks, Ratandeep!
[~vgarg] Will be good to have this in branch-3 as well.
> [Performance][Avro] Reading flat Avro tables is very expensive in Hive
> ----------------------------------------------------------------------
>
> Key: HIVE-18410
> URL: https://issues.apache.org/jira/browse/HIVE-18410
> Project: Hive
> Issue Type: Improvement
> Affects Versions: 1.2.1, 2.1.0, 3.0.0, 2.3.2
> Reporter: Ratandeep Ratti
> Assignee: Ratandeep Ratti
> Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-18410.patch, HIVE-18410_1.patch,
> HIVE-18410_2.patch, HIVE-18410_3.patch, profiling_with_patch.nps,
> profiling_with_patch.png, profiling_without_patch.nps,
> profiling_without_patch.png
>
>
> There's a performance penalty when reading flat [no nested fields] Avro
> tables. When reading the same flat dataset in Pig, it takes half the time.
> On profiling, a lot of time is spent in
> {{AvroDeserializer.deserializeSingleItemNullableUnion()}}. The bulk of the
> time is spent in GenericData.get().resolveUnion(), which calls
> GenericData.getSchemaName(Object datum), which does a lot of instanceof
> checks. This could be simplified with performance benefits. A approach is
> described in this patch which almost halves the runtime.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
