For some strange reason, neither hadoop-default.xml nor
hadoop-site.xml is loading. Both files are in hadoop-0.19.0/conf/
folder.
HADOOP_CONF_DIR points to thi
In some java code, (not a job), i did
Configuration cf = new Configuration();
String[] xu= new
String[]{"io.file.buffer.size","io.sort.mb","io.sort.factor","mapred.tasktracker.map.tasks.maximum","hadoop.logfile.count","hadoop.logfile.size","hadoop.tmp.dir","dfs.replication","mapred.map.tasks","mapred.job.tracker","fs.default.name"};
for(String x : xu){
LOG.info(x+": "+cf.get(x));
}
All the values returned were default values(as noted in
hadoop-default.xml). Changing the value in hadoop-default.xml were not
reflected here. Grepping the source reveal the defaults are
hardcoded. So it seems neither is hadoop-default.xml not
hadooop-site.xml loading.
Also, Configuration.java mentionds that hadoop-site.xml is depecated
(thought still loaded)
Any suggestions?
Regards
Saptarshi
On Mon, Jan 5, 2009 at 2:26 PM, Saptarshi Guha <[email protected]> wrote:
> Hello,
> I have set my HADOOP_CONF_DIR to the conf folder and still not
> loading. I have to manually set the options when I create my conf.
> Have you resolved this?
>
> Regards
> Saptarshi
>
> On Tue, Dec 30, 2008 at 5:25 PM, g00dn3ss <[email protected]> wrote:
>> Hey all,
>>
>> I have a similar issue. I am specifically having problems with the config
>> option "mapred.child.java.opts." I set it to -Xmx1024m and it uses -Xmx200m
>> regardless. I am running Hadoop 0.18.2 and I'm pretty sure this option was
>> working in the previous versions of Hadoop I was using.
>>
>> I am not explicitly setting HADOOP_CONF_DIR. My site config is in
>> ${HADOOP_HOME}/conf. Just to test things further, I wrote a small map task
>> to print out the ENV values and it has the correct value for HADOOP_HOME,
>> HADOOP_LOG_DIR, HADOOP_OPTS, etc... I also printed out the key/values in
>> the JobConf passed to the mapper and it has my specified values for
>> fs.default.name and mapred.job.tracker. Other settings like dfs.name.dir,
>> dfs.data.dir, and mapred.child.java.opts do not have my values.
>>
>> Any suggestion where to look at next?
>>
>> Thanks!
>>
>>
>>
>> On Mon, Dec 29, 2008 at 10:27 PM, Amareshwari Sriramadasu <
>> [email protected]> wrote:
>>
>>> Saptarshi Guha wrote:
>>>
>>>> Hello,
>>>> I had previously emailed regarding heap size issue and have discovered
>>>> that the hadoop-site.xml is not loading completely, i.e
>>>> Configuration defaults = new Configuration();
>>>> JobConf jobConf = new JobConf(defaults, XYZ.class);
>>>> System.out.println("1:"+jobConf.get("mapred.child.java.opts"));
>>>> System.out.println("2:"+jobConf.get("mapred.map.tasks"));
>>>> System.out.println("3:"+jobConf.get("mapred.reduce.tasks"));
>>>>
>>>>
>>>> System.out.println("3:"+jobConf.get("mapred.tasktracker.reduce.tasks.maximum"));
>>>>
>>>> returns -Xmx200m, 2,1,2 respectively, even though the numbers in the
>>>> hadoop-site.xml are very different.
>>>>
>>>> Is there a way for hadoop to dump the parameters read in from
>>>> hadoop-site.xml and hadoop-default.xml?
>>>>
>>>>
>>>>
>>> Is your hadoop-site.xml present in the conf (HADOOP_CONF_DIR) directory?
>>> http://hadoop.apache.org/core/docs/r0.19.0/cluster_setup.html#Configuration
>>>
>>> -Amareshwari
>>>
>>
>
>
>
> --
> Saptarshi Guha - [email protected]
>
--
Saptarshi Guha - [email protected]
