Hi All,
I'm trying to play with counters with PigServer and have a couple issues.
First, I've found very little documentation on how to do this, so I'm not
sure if the method I'm trying is the good one, any feedback would be
appreciated.
>From what I understand, we need a PigStats in order to be able to retrieve
the counters from it.
To get this PigStats from a PigServer instance, here is what I do :
pigServer.setBatchOn(); // needed to enable batch mode, which seems to be
the only way to get the ExecJob instances needed to get the stats
pigServer.registerScript(pigScript, params); // register the script i want
to run
List<ExecJob> execJobs = pigServer.executeBatch(); // get the ExecJobs
associated with the script i just ran
Now I am supposed to be able to get the counters from this ExecJob class.
for (ExecJob execJob : execJobs) {
for (JobStats jobStats : execJob.getStatistics().getJobGraph()) { // not
sure why we need to use the job graph to get the stats but that seems to be
the only solution i found
Counters counters = jobStats.getHadoopCounters(); // this is always
NULL !
for (Group group : counters) {
for (Counter counter : group) {
Now the strange thing is that every time I call the getHadoopCounters(),
the resulting Counters object is null, and thus I cannot get any counter at
all.
This happens in local and mapreduce mode, and I checked that execJob and
jobStats are indeed not null.
Am I doing something wrong here to get the counters, or forgetting
something? I'm using pig 0.8.1 from cdh3u1
Thanks for your help !
--
Charles Menguy | Senior Software Engineer
Proclivity Systems
22 West 19th Street | 9th Floor
New York, NY 10011
[email protected]
www.proclivitysystems.com
Proclivity® | We Value Your Customers™
This message is the property of Proclivity Systems, Inc. and is intended
only for the use of the addressee(s), and may contain material that is
confidential and privileged for the sole use of the intended recipient. If
you are not the intended recipient, reliance or forwarding without express
permission is strictly prohibited; please contact the sender and delete all
copies.
