There is a good blog article on this- http://squarecog.wordpress.com/2010/12/24/incrementing-hadoop-counters-in-apache-pig/
Thanks, Aniket On Tue, Dec 6, 2011 at 1:49 PM, Charles Menguy < [email protected]> wrote: > Hi All, > > I'm trying to play with counters with PigServer and have a couple issues. > > First, I've found very little documentation on how to do this, so I'm not > sure if the method I'm trying is the good one, any feedback would be > appreciated. > > From what I understand, we need a PigStats in order to be able to retrieve > the counters from it. > To get this PigStats from a PigServer instance, here is what I do : > > pigServer.setBatchOn(); // needed to enable batch mode, which seems to be > the only way to get the ExecJob instances needed to get the stats > pigServer.registerScript(pigScript, params); // register the script i want > to run > List<ExecJob> execJobs = pigServer.executeBatch(); // get the ExecJobs > associated with the script i just ran > > Now I am supposed to be able to get the counters from this ExecJob class. > > for (ExecJob execJob : execJobs) { > for (JobStats jobStats : execJob.getStatistics().getJobGraph()) { // not > sure why we need to use the job graph to get the stats but that seems to be > the only solution i found > Counters counters = jobStats.getHadoopCounters(); // this is always > NULL ! > for (Group group : counters) { > for (Counter counter : group) { > > Now the strange thing is that every time I call the getHadoopCounters(), > the resulting Counters object is null, and thus I cannot get any counter at > all. > > This happens in local and mapreduce mode, and I checked that execJob and > jobStats are indeed not null. > > Am I doing something wrong here to get the counters, or forgetting > something? I'm using pig 0.8.1 from cdh3u1 > > Thanks for your help ! > > -- > Charles Menguy | Senior Software Engineer > Proclivity Systems > 22 West 19th Street | 9th Floor > New York, NY 10011 > [email protected] > www.proclivitysystems.com > > Proclivity® | We Value Your Customers™ > > This message is the property of Proclivity Systems, Inc. and is intended > only for the use of the addressee(s), and may contain material that is > confidential and privileged for the sole use of the intended recipient. If > you are not the intended recipient, reliance or forwarding without express > permission is strictly prohibited; please contact the sender and delete all > copies. > -- "...:::Aniket:::... Quetzalco@tl"
