[
https://issues.apache.org/jira/browse/HIVE-14887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15613093#comment-15613093
]
Siddharth Seth commented on HIVE-14887:
---------------------------------------
Thanks for the review. Committing.
In terms of #of HDFS nodes. Each node is launched as a thread within the same
process (the process launched by maven to execute tests). That's currently 2GB,
and I'll file a separate jira to reduce this after this one is committed. When
moving this to 1GB - I didn't really see any failures related to HDFS itself.
A 4 node MiniYARNCluster runs all 4 nodes as threads as well. However, the
difference is that the containers that run on the NodeManagers launch as
separate processes. Without this patch - we would run 4 NMs, with 4 1GB
containers each - so each test could end up running with 16GB. Post the changes
there's 2 NMs, with 4x128MB. So this gets way lower.
> Reduce the memory requirements for tests
> ----------------------------------------
>
> Key: HIVE-14887
> URL: https://issues.apache.org/jira/browse/HIVE-14887
> Project: Hive
> Issue Type: Sub-task
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Attachments: HIVE-14887.01.patch, HIVE-14887.02.patch,
> HIVE-14887.03.patch, HIVE-14887.04.patch, HIVE-14887.05.patch,
> HIVE-14887.06.patch
>
>
> The clusters that we spin up end up requiring 16GB at times. Also the maven
> arguments seem a little heavy weight.
> Reducing this will allow for additional ptest drones per box, which should
> bring down the runtime.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
