Matthias Jacob created HADOOP-12490:
---------------------------------------
Summary: Add default resources to Configuration which are not
inside the class path
Key: HADOOP-12490
URL: https://issues.apache.org/jira/browse/HADOOP-12490
Project: Hadoop Common
Issue Type: Improvement
Components: conf
Reporter: Matthias Jacob
*Problem*
It is currently not possible to add default resources to
{{org.apache.hadoop.conf.Configuration}} that are not included in the class
path.
We need to add resources to {{org.apache.hadoop.conf.Configuration}} that are
not included in the class path. Currently, the only way to do this is to use
{{addResource(URL)}} or {{addResource(Path)}} which both are instance methods.
However, a library we work with creates Configuration instances internally that
we cannot access. So we need to add them as default resources in order to make
them available to all subsequent Configuration instances (and those existing
instances where {{loadDefaults}} equals {{true}}, of course).
*Solution*
The proposed solution is to add the following static methods:
{code}
public static synchronized void addDefaultResource(URL url);
public static synchronized void addDefaultResource(Path file);
{code}
Please see the attached patch for details.
*Example*
In order to initialize the Configuration correctly, what we need to do
currently is as follows:
{code}
// Set resource paths for the class loader
File ffile = new File("/path/to/conf/dir");
URL url = ffile.toURI().toURL();
URLClassLoader systemClassLoader = (URLClassLoader) getClass().getClassLoader();
Class<URLClassLoader> classLoaderClass = URLClassLoader.class;
Method method = classLoaderClass.getDeclaredMethod("addURL", new
Class[]{URL.class});
method.setAccessible(true);
method.invoke(systemClassLoader, new Object[]{url});
//-----------------------------
// Read cluster config
for (File f : ffile.listFiles()) {
if(f.getName().toLowerCase().endsWith(".xml")) {
Configuration.addDefaultResource(f.getName());
}
}
{code}
As you can see, we currently need to manipulate the class path through
reflection which is kind of dirty.
With the proposed solution, the same (adding default resources from outside the
class path) can be achieved doing the following:
{code}
// Set resource paths for the class loader
File ffile = new File("/path/to/conf/dir");
//-----------------------------
// Read cluster config
for (File f : ffile.listFiles()) {
if(f.getName().toLowerCase().endsWith(".xml")) {
Configuration.addDefaultResource(new Path(f.getAbsolutePath()));
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
