Kevin Wilfong created HIVE-4157:
-----------------------------------
Summary: ORC runs out of heap when writing
Key: HIVE-4157
URL: https://issues.apache.org/jira/browse/HIVE-4157
Project: Hive
Issue Type: Improvement
Components: Serializers/Deserializers
Affects Versions: 0.11.0
Reporter: Kevin Wilfong
Assignee: Kevin Wilfong
The OutStream class used by the ORC file format seems to aggressively allocate
memory for ByteBuffers and doesn't seem too eager to give it back.
This causes issues with heap space, particularly when a wide tables/dynamic
partitions are involved.
As a first step to resolving this problem, the OutStream class can be modified
to lazily allocate memory, and more actively make it available for garbage
collection.
Follow ups could include checking the amount of free memory as part of
determining if a spill is needed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
