[
https://issues.apache.org/jira/browse/HBASE-16375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15448627#comment-15448627
]
Loknath Priyatham Teja Singamsetty commented on HBASE-16375:
-------------------------------------------------------------
[~busbey] [~apurtell] I see that there is TestHBaseTestingUtility.java. However
there are no tests which are written specially to check the conf being set
already.
{quote}
String rmAddress = jobConf.get("yarn.resourcemanager.address");
if (rmAddress != null) {
conf.set("yarn.resourcemanager.address", rmAddress);
}
String historyAddress = jobConf.get("mapreduce.jobhistory.address");
if (historyAddress != null) {
conf.set("mapreduce.jobhistory.address", historyAddress);
}
String schedulerAddress =
jobConf.get("yarn.resourcemanager.scheduler.address");
if (schedulerAddress != null) {
conf.set("yarn.resourcemanager.scheduler.address", schedulerAddress);
}
{quote}
newly added configs
{quote}
String mrJobHistoryWebappAddress =
+ jobConf.get("mapreduce.jobhistory.webapp.address");
+ if (mrJobHistoryWebappAddress != null) {
+ conf.set("mapreduce.jobhistory.webapp.address",
mrJobHistoryWebappAddress);
+ }
+ String yarnRMWebappAddress =
+ jobConf.get("yarn.resourcemanager.webapp.address");
+ if (yarnRMWebappAddress != null) {
+ conf.set("yarn.resourcemanager.webapp.address", yarnRMWebappAddress);
+ }
{quote}
If we need code coverage test on the configs, then we need to write a unit test
that starts mini mapreduce cluster and verifies that the jobConf from
MapReduceTestingShim are being populated to instance variable "conf" of
HBaseTestingUtility.
Are we looking for such a test to be added here (to cover existing and newly
added configs)?
> Mapreduce mini cluster using HBaseTestingUtility not setting correct
> resourcemanager and jobhistory webapp address of MapReduceTestingShim
> --------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-16375
> URL: https://issues.apache.org/jira/browse/HBASE-16375
> Project: HBase
> Issue Type: Bug
> Affects Versions: 1.2.0
> Reporter: Loknath Priyatham Teja Singamsetty
> Assignee: Loknath Priyatham Teja Singamsetty
> Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.1.6, 0.98.22, 1.2.4
>
> Attachments: HBASE-16375_0.98_and_above.patch
>
>
> Starting mapreduce mini cluster using HBaseTestingUtility is not setting
> "yarn.resourcemanager.webapp.address" and
> "mapreduce.jobhistory.webapp.address" which are required for getting the
> submitted yarn apps using mapreduce webapp. These properties are not being
> copied from jobConf of MapReduceTestingShim resulting in default values.
> {quote}
> HBaseTestingUtility.java
> // Allow the user to override FS URI for this map-reduce cluster to use.
> mrCluster = new MiniMRCluster(servers,
> FS_URI != null ? FS_URI : FileSystem.get(conf).getUri().toString(), 1,
> null, null, new JobConf(this.conf));
> JobConf jobConf = MapreduceTestingShim.getJobConf(mrCluster);
> if (jobConf == null) {
> jobConf = mrCluster.createJobConf();
> }
> jobConf.set("mapreduce.cluster.local.dir",
> conf.get("mapreduce.cluster.local.dir")); //Hadoop MiniMR overwrites
> this while it should not
> LOG.info("Mini mapreduce cluster started");
> // In hadoop2, YARN/MR2 starts a mini cluster with its own conf instance
> and updates settings.
> // Our HBase MR jobs need several of these settings in order to properly
> run. So we copy the
> // necessary config properties here. YARN-129 required adding a few
> properties.
> conf.set("mapreduce.jobtracker.address",
> jobConf.get("mapreduce.jobtracker.address"));
> // this for mrv2 support; mr1 ignores this
> conf.set("mapreduce.framework.name", "yarn");
> conf.setBoolean("yarn.is.minicluster", true);
> String rmAddress = jobConf.get("yarn.resourcemanager.address");
> if (rmAddress != null) {
> conf.set("yarn.resourcemanager.address", rmAddress);
> }
> String historyAddress = jobConf.get("mapreduce.jobhistory.address");
> if (historyAddress != null) {
> conf.set("mapreduce.jobhistory.address", historyAddress);
> }
> String schedulerAddress =
> jobConf.get("yarn.resourcemanager.scheduler.address");
> if (schedulerAddress != null) {
> conf.set("yarn.resourcemanager.scheduler.address", schedulerAddress);
> }
> {quote}
> As a immediate fix for phoenix e2e test to succeed, need the below lines to
> be added as well
> {quote}
> String rmWebappAddress =
> jobConf.get("yarn.resourcemanager.webapp.address");
> if (rmWebappAddress != null) {
> conf.set("yarn.resourcemanager.webapp.address", rmWebappAddress);
> }
> String historyWebappAddress =
> jobConf.get("mapreduce.jobhistory.webapp.address");
> if (historyWebappAddress != null) {
> conf.set("mapreduce.jobhistory.webapp.address", historyWebappAddress);
> }
> {quote}
> Eventually, we should also see if we can copy over all the jobConf properties
> to HBaseTestingUtility conf object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
