[
https://issues.apache.org/jira/browse/HIVE-1511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13761526#comment-13761526
]
Leo Romanoff commented on HIVE-1511:
------------------------------------
It is very cool that you got so far with Kryo! Could you report about the
performance of this new Kryo-based approach vs the old one based on XML? Is it
improved? How much faster/slower is it?
If there is a need to improve it further, there are some further tricks that
could be used to make Kryo even faster. You already reuse ThreadLocal Kryo
instances which is good. But to further improve performance you could:
1) Start using Unsafe-based streams (e.g. UnsafeInput and UnsafeOutput. Or
their direct-memory based versions, e.g.in case you want to write directly into
off-heap memory). This can improve performance very significantly (e.g.
2X-30X).
2) Pre-register all of the classes. This may reduce the size of serialized
representation and also improves performance a bi.
-Leo
> Hive plan serialization is slow
> -------------------------------
>
> Key: HIVE-1511
> URL: https://issues.apache.org/jira/browse/HIVE-1511
> Project: Hive
> Issue Type: Improvement
> Affects Versions: 0.7.0, 0.11.0
> Reporter: Ning Zhang
> Assignee: Mohammad Kamrul Islam
> Attachments: failedPlan.xml, generated_plan.xml, HIVE-1511.10.patch,
> HIVE-1511.11.patch, HIVE-1511.12.patch, HIVE-1511.13.patch,
> HIVE-1511.14.patch, HIVE-1511.16.patch, HIVE-1511.17.patch,
> HIVE-1511.4.patch, HIVE-1511.5.patch, HIVE-1511.6.patch, HIVE-1511.7.patch,
> HIVE-1511.8.patch, HIVE-1511.9.patch, HIVE-1511.patch, HIVE-1511-wip2.patch,
> HIVE-1511-wip3.patch, HIVE-1511-wip4.patch, HIVE-1511.wip.9.patch,
> HIVE-1511-wip.patch, KryoHiveTest.java, run.sh
>
>
> As reported by Edward Capriolo:
> For reference I did this as a test case....
> SELECT * FROM src where
> key=0 OR key=0 OR key=0 OR key=0 OR key=0 OR key=0 OR key=0 OR key=0
> OR key=0 OR key=0 OR key=0 OR
> key=0 OR key=0 OR key=0 OR key=0 OR key=0 OR key=0 OR key=0 OR key=0
> OR key=0 OR key=0 OR key=0 OR
> ...(100 more of these)
> No OOM but I gave up after the test case did not go anywhere for about
> 2 minutes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
