You can have your item in a separate jar and pin the reference so that it
becomes perm-gen, which will pin it. Then you can search the class loader
hierarchy for the reference.
A quick scan through the Child.java main loop shows no magic with class
loaders.
I wrote some code to check this against 0.19.0. Very clearly the JVM is
doing nothing special with class loaders.
The classes are loaded exactly once.
The job counters contain details of how many times the map method was called
and the number of times the singleton was taken, including the jvm pid's
I wrote a mapper and a singleton, the singleton has 1 method that returns
the number of times that the getSingleton
The code fragment is from the examples in my book
http://www.apress.com/book/view/9781430219422
On Fri, Mar 6, 2009 at 12:55 PM, Scott Carey <[email protected]>wrote:
> One further thought on this, the mapper jvm may be loading the jar and
> overwriting / throwing away all previous class descriptions from the
> previous map job, which will remove the statics and reinitialize. In this
> case, the singleton won't work if it is in the job jar. What will work is
> putting the singleton in a global classpath (shared library not in the job
> jar).
>
>
> On 3/6/09 12:46 PM, "Scott Carey" <[email protected]> wrote:
>
> The difference is that if the whole mapper class itself is being reloaded
> somehow (instantiated by reflection and then de-referenced and gc'd?) the
> static won't work the way you expect. Not knowing how that works, and
> assuming that statics there don't work, a singleton in another class may
> still work. The singleton class is certainly not being instantiated by
> reflection so (I believe) only a classloader closing will get rid of it.
>
> At least, its worth a try, since unlike the mapper class, you control how
> it is instantiated. So the two cases are not the same.
>
>
> On 3/6/09 12:13 AM, "Rasit OZDAS" <[email protected]> wrote:
>
> Owen, I tried this, it doesn't work.
> I doubt if static singleton method will work either,
> since it's much or less the same.
>
> Rasit
>
> 2009/3/2 Owen O'Malley <[email protected]>
>
> >
> > On Mar 2, 2009, at 3:03 AM, Tom White wrote:
> >
> > I believe the static singleton approach outlined by Scott will work
> >> since the map classes are in a single classloader (but I haven't
> >> actually tried this).
> >>
> >
> > Even easier, you should just be able to do it with static initialization
> in
> > the Mapper class. (I haven't tried it either... )
> >
> > -- Owen
> >
>
>
>
> --
> M. Raşit ÖZDAŞ
>
>
>
The Job is complete and successfull
Counter Group: File Systems
HDFS bytes read 11539
HDFS bytes written 1116
Counter Group: Input
Total 24
Counter Group: Job Counters
Launched map tasks 24
Counter Group: Output
Total 24
/** Each group here is named for the jvm pid and hostname, and clearly shows
the number of tasks run per jvm */
Counter Group: 3...@host1
attempt_200902221346_0096_m_000014_0 1
attempt_200902221346_0096_m_000018_0 1
attempt_200902221346_0096_m_000021_0 1
Counter Group: 16...@host2
attempt_200902221346_0096_m_000013_0 1
attempt_200902221346_0096_m_000009_0 1
attempt_200902221346_0096_m_000006_0 1
attempt_200902221346_0096_m_000017_0 1
attempt_200902221346_0096_m_000023_0 1
Counter Group: 2...@host1
attempt_200902221346_0096_m_000012_0 1
attempt_200902221346_0096_m_000005_0 1
attempt_200902221346_0096_m_000000_0 1
attempt_200902221346_0096_m_000001_0 1
attempt_200902221346_0096_m_000010_0 1
attempt_200902221346_0096_m_000007_0 1
Counter Group: 16...@host2
attempt_200902221346_0096_m_000019_0 1
attempt_200902221346_0096_m_000015_0 1
attempt_200902221346_0096_m_000008_0 1
attempt_200902221346_0096_m_000003_0 1
attempt_200902221346_0096_m_000011_0 1
Counter Group: 2...@host1
attempt_200902221346_0096_m_000022_0 1
attempt_200902221346_0096_m_000002_0 1
attempt_200902221346_0096_m_000020_0 1
attempt_200902221346_0096_m_000016_0 1
attempt_200902221346_0096_m_000004_0 1
/** This shows the number of times the map method was called in each task. */
Counter Group: Tasks
attempt_200902221346_0096_m_000014_0 1
attempt_200902221346_0096_m_000012_0 1
attempt_200902221346_0096_m_000015_0 1
attempt_200902221346_0096_m_000009_0 1
attempt_200902221346_0096_m_000006_0 1
attempt_200902221346_0096_m_000022_0 1
attempt_200902221346_0096_m_000021_0 1
attempt_200902221346_0096_m_000020_0 1
attempt_200902221346_0096_m_000001_0 1
attempt_200902221346_0096_m_000003_0 1
attempt_200902221346_0096_m_000017_0 1
attempt_200902221346_0096_m_000004_0 1
attempt_200902221346_0096_m_000019_0 1
attempt_200902221346_0096_m_000013_0 1
attempt_200902221346_0096_m_000005_0 1
attempt_200902221346_0096_m_000000_0 1
attempt_200902221346_0096_m_000018_0 1
attempt_200902221346_0096_m_000002_0 1
attempt_200902221346_0096_m_000008_0 1
attempt_200902221346_0096_m_000010_0 1
attempt_200902221346_0096_m_000016_0 1
attempt_200902221346_0096_m_000007_0 1
attempt_200902221346_0096_m_000023_0 1
attempt_200902221346_0096_m_000011_0 1
/** This shows the number of times the singleton has been requested, in a given
task */
Counter Group: Singleton
attempt_200902221346_0096_m_000014_0 1
attempt_200902221346_0096_m_000012_0 6
attempt_200902221346_0096_m_000015_0 4
attempt_200902221346_0096_m_000009_0 2
attempt_200902221346_0096_m_000006_0 1
attempt_200902221346_0096_m_000022_0 5
attempt_200902221346_0096_m_000021_0 3
attempt_200902221346_0096_m_000020_0 4
attempt_200902221346_0096_m_000001_0 2
attempt_200902221346_0096_m_000003_0 1
attempt_200902221346_0096_m_000017_0 4
attempt_200902221346_0096_m_000004_0 2
attempt_200902221346_0096_m_000019_0 5
attempt_200902221346_0096_m_000013_0 3
attempt_200902221346_0096_m_000005_0 3
attempt_200902221346_0096_m_000000_0 1
attempt_200902221346_0096_m_000018_0 2
attempt_200902221346_0096_m_000002_0 1
attempt_200902221346_0096_m_000008_0 2
attempt_200902221346_0096_m_000010_0 5
attempt_200902221346_0096_m_000016_0 3
attempt_200902221346_0096_m_000007_0 4
attempt_200902221346_0096_m_000023_0 5
attempt_200902221346_0096_m_000011_0 3
Counter Group: Map-Reduce Framework
Map input records 24
Map input bytes 316
Map output records 24
// Code Fragment from the chapter on Advanced Techniques in the book ProHadoop
by Apress.com
/** This class just keeps a count of the number of times the singleton
was gotten through the factory. */
public static class Singleton {
/** Our singleton. */
public static Singleton singleton = null;
/** counter of the number of times the singleton has been
requested. */
static AtomicLong getCounter = new AtomicLong(0);
static {
/** Construct the singleton here since we know we are
going to us it. */
singleton = new Singleton();
}
protected Singleton() {
}
/** Get an instance of the singleton, and increment our static
counter. *
*
* @return the singleton
*/
public static Singleton getSingleton() {
getCounter.incrementAndGet();
return singleton;
}
/** Return the number of times the singleton has been
requested. */
public long getCurrentCounter() {
return getCounter.get();
}
}
/**
* @author Jason
*
*/
public static class TestMapper extends MapReduceBase implements
Mapper<LongWritable, Text, Text, LongWritable> {
Singleton singleton;
String taskName;
TaskAttemptID taskId;
/** grab our singleton and the taskid. */
@Override
public void configure(JobConf conf) {
taskName = conf.getJobName();
taskId =
TaskAttemptID.forName(conf.get("mapred.task.id"));
if (taskName == null || taskName.length() == 0) {
/** if the job name is essentially unset make
something up. */
taskName = taskId.isMap() ? "map." : "reduce."
+ this.getClass().getName();
}
singleton = Singleton.getSingleton();
}
/** assume there is only 1 key per map, and report on
information about the singleton state. */
@Override
public void map(LongWritable key, Text value,
OutputCollector<Text, LongWritable> output,
Reporter reporter)
throws IOException {
/** Our key is the line number, which should be 1, and
the single line, which should be the file name. */
try {
reporter.incrCounter("Input", "Total", 1);
long initCount = singleton.getCurrentCounter();
reporter.incrCounter("Singleton",
taskId.toString(), initCount);
reporter.incrCounter("Tasks",
taskId.toString(), 1);
key.set(initCount);
output.collect(value, key);
reporter.incrCounter("Output", "Total", 1);
reporter.incrCounter(ManagementFactory.getRuntimeMXBean().getName(),
taskId.toString(), 1);
} catch (Throwable e) {
reporter.incrCounter("Exceptions", "Total", 1);
reporter.incrCounter("Exceptions",
e.getClass().getName(), 1);
if (e instanceof RuntimeException) {
throw (RuntimeException) e;
}
if (e instanceof IOException) {
throw (IOException) e;
}
throw new IOException(e);
}
}
}
Logger LOG = Logger.getLogger(JVMReuseAndStaticInitializers.class);
/** This is where derived classes define required setup.
* @throws IOException
*/
@Override
protected
void customSetup(JobConf conf) throws IOException {
super.customSetup(conf);
/** Setup for NLineInputFormat, which has LongWritable, Text
key, values, and only 1 input line per map. */
conf.setInputFormat(NLineInputFormat.class);
conf.setInt("mapred.line.input.format.linespermap",1);
/** Work out how many map slots there are in the cluster. */
JobClient client = new JobClient(conf);
ClusterStatus status = client.getClusterStatus();
int mapSlots = status.getMaxMapTasks();
if (verbose) { LOG.info("There are " + mapSlots + " map
execution slots in this cluster, setting up for " + mapSlots * 6 + "inputs"); }
/** Setup our input so we have 6 input records per map slot. */
Path root = new Path("JVMReuseAndStaticInitializers");
Utils.makeSampleInputIf(conf, root.toString(), mapSlots*6);
FileInputFormat.setInputPaths(conf, root);
conf.setMapperClass(TestMapper.class);
/** Ensure that each map is run one time only, to avoid biasing
the singleton counts per taskid. */
conf.setMapSpeculativeExecution(false);
conf.setMaxMapAttempts(1);
/** Setup some jvm reuse. */
conf.setNumTasksToExecutePerJvm(6);
FileOutputFormat.setOutputPath(conf, root.suffix(".output"));
conf.setOutputKeyClass(Text.class);
conf.setOutputValueClass(LongWritable.class);
conf.setNumReduceTasks(0);
conf.set("mapred.child.java.opts", "-Xmx200m -verbose:class");
}
