[
https://issues.apache.org/jira/browse/HIVE-16758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16115918#comment-16115918
]
BELUGA BEHR commented on HIVE-16758:
------------------------------------
[~csun] Ah, yes. I see. The default value of
{{mapreduce.client.submit.file.replication}} is 10. Perhaps this is where the
original author derived the existing default value from. Yes, if this was set
to a default of 10, I think the job would fail and not implicitly be adjusted
to a factor of 3. That's the problem that lead me to this patch in the first
place. However, if it is the case that it will fail, then the Spark Job would
fail even before getting to this point because the initial Spark Job JAR upload
to HDFS would fail during the initial staging phase, before execution.
>From my limited experience, my expectation is that this would improve the
>performance of a larger cluster.
The values 10, {{dfs.replication.max}}, and {{getDefaultReplication}} are not
good proxies for determining cluster size. The setting
{{mapreduce.client.submit.file.replication}} is expected to be set appropriate
value based on the cluster size.
Please consider a cluster with 200 nodes, a {{dfs.replication.max}} of 100, and
a default replication of 3.
{code}
minReplication = Math.min(minReplication, dfsMaxReplication);
minReplication = Math.min(10, 200);
minReplication = 10;
replication = Math.max(minReplication, defaultReplication);
replication = Math.max(10, 3);
replication = 10;
{code}
So, for a 200 node cluster, only 10 nodes would contain this data. The
{{mapreduce.client.submit.file.replication}} should be set to _sqrt(200)_ = 14.
For larger clusters, then the difference between _10_ and the recommended
amount would be even greater: _sqrt(300)_ _sqrt(400)_ _sqrt(500)_, etc.
> Better Select Number of Replications
> ------------------------------------
>
> Key: HIVE-16758
> URL: https://issues.apache.org/jira/browse/HIVE-16758
> Project: Hive
> Issue Type: Improvement
> Reporter: BELUGA BEHR
> Assignee: BELUGA BEHR
> Priority: Minor
> Attachments: HIVE-16758.1.patch
>
>
> {{org.apache.hadoop.hive.ql.exec.SparkHashTableSinkOperator.java}}
> We should be smarter about how we pick a replication number. We should add a
> new configuration equivalent to {{mapreduce.client.submit.file.replication}}.
> This value should be around the square root of the number of nodes and not
> hard-coded in the code.
> {code}
> public static final String DFS_REPLICATION_MAX = "dfs.replication.max";
> private int minReplication = 10;
> @Override
> protected void initializeOp(Configuration hconf) throws HiveException {
> ...
> int dfsMaxReplication = hconf.getInt(DFS_REPLICATION_MAX, minReplication);
> // minReplication value should not cross the value of dfs.replication.max
> minReplication = Math.min(minReplication, dfsMaxReplication);
> }
> {code}
> https://hadoop.apache.org/docs/r2.7.2/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
