Hi Russell, I'm a newbie to Pig, and I was instructed to downgrade to Pig 0.13 as the Pig 0.14 has issues with respect to jar files. I am on Windows 7. I did so, but still I face exceptions while running Pig 0.13.
You have said that you have been working with Pig for 5 years. Is it possible for you to take a look at my logs (Pig 0.13) and tell me what am I doing wrong? If you can, I will send you the logs. Thanks in advance, Venkat. On Thu, Jan 22, 2015 at 11:36 AM, Russell Jurney <[email protected]> wrote: > Now I installed Hadoop 2.6, and I compiled both Pig and piggybank using > 'ant -Dhadoopversion=23 jar' > > Still doesn't work... > > > Russells-MacBook-Pro-OLD:java rjurney$ pig -l /tmp -v -w -x local > > 2015-01-21 22:04:25.808 java[34529:1903] Unable to load realm info from > SCDynamicStore > > 15/01/21 22:04:26 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL > > 15/01/21 22:04:26 INFO pig.ExecTypeProvider: Picked LOCAL as the ExecType > > 2015-01-21 22:04:26,120 [main] INFO org.apache.pig.Main - Apache Pig > version 0.14.0-SNAPSHOT (rUnversioned directory) compiled Jan 21 2015, > 22:01:34 > > 2015-01-21 22:04:26,120 [main] INFO org.apache.pig.Main - Logging error > messages to: /private/tmp/pig_1421906665386.log > > 2015-01-21 22:04:26,335 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is > deprecated. Instead, use fs.defaultFS > > 2015-01-21 22:04:26,335 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is > deprecated. Instead, use mapreduce.jobtracker.address > > 2015-01-21 22:04:26,337 [main] INFO > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting > to hadoop file system at: file:/// > > 2015-01-21 22:04:26,439 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - io.bytes.per.checksum is > deprecated. Instead, use dfs.bytes-per-checksum > > grunt> REGISTER > /Users/rjurney/Software/pig-0.14.0/contrib/piggybank/java/piggybank.jar > > 2015-01-21 22:04:28,665 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is > deprecated. Instead, use fs.defaultFS > > 2015-01-21 22:04:28,668 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - io.bytes.per.checksum is > deprecated. Instead, use dfs.bytes-per-checksum > > grunt> > > grunt> data = LOAD '/Users/rjurney/Software/atomic_nile/data/gsa_feed.xml' > USING org.apache.pig.piggybank.storage.XMLLoader('record') AS > (doc:chararray); > > 2015-01-21 22:04:29,652 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is > deprecated. Instead, use fs.defaultFS > > 2015-01-21 22:04:29,654 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - io.bytes.per.checksum is > deprecated. Instead, use dfs.bytes-per-checksum > > grunt> dump data > > 2015-01-21 22:04:30,940 [main] WARN > org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > > 2015-01-21 22:04:30,989 [main] INFO > org.apache.pig.tools.pigstats.ScriptState - Pig features used in the > script: UNKNOWN > > 2015-01-21 22:04:31,040 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - io.bytes.per.checksum is > deprecated. Instead, use dfs.bytes-per-checksum > > 2015-01-21 22:04:31,042 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is > deprecated. Instead, use fs.defaultFS > > 2015-01-21 22:04:31,087 [main] INFO > org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer - > {RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, ConstantCalculator, > GroupByConstParallelSetter, LimitOptimizer, LoadTypeCastInserter, > MergeFilter, MergeForEach, PartitionFilterOptimizer, > PredicatePushdownOptimizer, PushDownForEachFlatten, PushUpFilter, > SplitFilter, StreamTypeCastInserter]} > > 2015-01-21 22:04:31,209 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - > File concatenation threshold: 100 optimistic? false > > 2015-01-21 22:04:31,235 [main] INFO > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer > - MR plan size before optimization: 1 > > 2015-01-21 22:04:31,235 [main] INFO > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer > - MR plan size after optimization: 1 > > 2015-01-21 22:04:31,268 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - io.bytes.per.checksum is > deprecated. Instead, use dfs.bytes-per-checksum > > 2015-01-21 22:04:31,270 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is > deprecated. Instead, use fs.defaultFS > > 2015-01-21 22:04:31,285 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - session.id is > deprecated. Instead, use dfs.metrics.session-id > > 2015-01-21 22:04:31,286 [main] INFO > org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics with > processName=JobTracker, sessionId= > > 2015-01-21 22:04:31,322 [main] WARN > org.apache.pig.backend.hadoop20.PigJobControl - falling back to default > JobControl (not using hadoop 0.20 ?) > > java.lang.NoSuchFieldException: runnerState > > at java.lang.Class.getDeclaredField(Class.java:1938) > > at > > org.apache.pig.backend.hadoop20.PigJobControl.<clinit>(PigJobControl.java:51) > > at > > org.apache.pig.backend.hadoop.executionengine.shims.HadoopShims.newJobControl(HadoopShims.java:109) > > at > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:314) > > at > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:196) > > at > > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:280) > > at org.apache.pig.PigServer.launchPlan(PigServer.java:1390) > > at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1375) > > at org.apache.pig.PigServer.storeEx(PigServer.java:1034) > > at org.apache.pig.PigServer.store(PigServer.java:997) > > at org.apache.pig.PigServer.openIterator(PigServer.java:910) > > at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:746) > > at > > org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:372) > > at > > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:230) > > at > > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205) > > at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:66) > > at org.apache.pig.Main.run(Main.java:558) > > at org.apache.pig.Main.main(Main.java:170) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:606) > > at org.apache.hadoop.util.RunJar.run(RunJar.java:221) > > at org.apache.hadoop.util.RunJar.main(RunJar.java:136) > > 2015-01-21 22:04:31,326 [main] INFO > org.apache.pig.tools.pigstats.mapreduce.MRScriptState - Pig script settings > are added to the job > > 2015-01-21 22:04:31,331 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.job.reduce.markreset.buffer.percent is deprecated. Instead, use > mapreduce.reduce.markreset.buffer.percent > > 2015-01-21 22:04:31,331 [main] INFO > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler > - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3 > > 2015-01-21 22:04:31,331 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - mapred.output.compress > is deprecated. Instead, use mapreduce.output.fileoutputformat.compress > > 2015-01-21 22:04:31,364 [main] INFO > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler > - Setting up single store job > > 2015-01-21 22:04:31,374 [main] INFO > org.apache.pig.data.SchemaTupleFrontend - Key [pig.schematuple] is false, > will not generate code. > > 2015-01-21 22:04:31,374 [main] INFO > org.apache.pig.data.SchemaTupleFrontend - Starting process to move > generated code to distributed cacche > > 2015-01-21 22:04:31,374 [main] INFO > org.apache.pig.data.SchemaTupleFrontend - Distributed cache not supported > or needed in local mode. Setting key [pig.schematuple.local.dir] with code > temp directory: > /var/folders/0b/74l_65015_5fcbmbdz1w2xl40000gn/T/1421906671373-0 > > 2015-01-21 22:04:31,406 [main] INFO > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - 1 map-reduce job(s) waiting for submission. > > 2015-01-21 22:04:31,407 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.job.tracker.http.address is deprecated. Instead, use > mapreduce.jobtracker.http.address > > 2015-01-21 22:04:31,417 [JobControl] INFO > org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics > with processName=JobTracker, sessionId= - already initialized > > 2015-01-21 22:04:31,428 [JobControl] ERROR > org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl - Error while trying > to run jobs. > > java.lang.IncompatibleClassChangeError: Found interface > org.apache.hadoop.mapreduce.JobContext, but class was expected > > at > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setupUdfEnvAndStores(PigOutputFormat.java:235) > > at > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.checkOutputSpecs(PigOutputFormat.java:183) > > at > org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:562) > > at > > org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:432) > > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296) > > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1293) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAs(Subject.java:415) > > at > > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) > > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293) > > at > > org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335) > > at > > org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:240) > > at > org.apache.pig.backend.hadoop20.PigJobControl.run(PigJobControl.java:121) > > at java.lang.Thread.run(Thread.java:724) > > at > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:276) > > 2015-01-21 22:04:31,429 [main] INFO > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - 0% complete > > 2015-01-21 22:04:31,435 [main] WARN > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to > stop immediately on failure. > > 2015-01-21 22:04:31,435 [main] INFO > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - job null has failed! Stop running all dependent jobs > > 2015-01-21 22:04:31,435 [main] INFO > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - 100% complete > > 2015-01-21 22:04:31,442 [main] ERROR > org.apache.pig.tools.pigstats.mapreduce.MRPigStatsUtil - 1 map reduce > job(s) failed! > > 2015-01-21 22:04:31,443 [main] INFO > org.apache.pig.tools.pigstats.mapreduce.SimplePigStats - Script Statistics: > > > HadoopVersion PigVersion UserId StartedAt FinishedAt Features > > 2.6.0 0.14.0-SNAPSHOT rjurney 2015-01-21 22:04:31 2015-01-21 22:04:31 > UNKNOWN > > > Failed! > > > Failed Jobs: > > JobId Alias Feature Message Outputs > > N/A data MAP_ONLY Message: Unexpected System Error Occured: > java.lang.IncompatibleClassChangeError: Found interface > org.apache.hadoop.mapreduce.JobContext, but class was expected > > at > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setupUdfEnvAndStores(PigOutputFormat.java:235) > > at > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.checkOutputSpecs(PigOutputFormat.java:183) > > at > org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:562) > > at > > org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:432) > > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296) > > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1293) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAs(Subject.java:415) > > at > > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) > > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293) > > at > > org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335) > > at > > org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:240) > > at > org.apache.pig.backend.hadoop20.PigJobControl.run(PigJobControl.java:121) > > at java.lang.Thread.run(Thread.java:724) > > at > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:276) > > file:/tmp/temp1277658924/tmp730730286, > > > Input(s): > > Failed to read data from > "/Users/rjurney/Software/atomic_nile/data/gsa_feed.xml" > > > Output(s): > > Failed to produce result in "file:/tmp/temp1277658924/tmp730730286" > > > Counters: > > Total records written : 0 > > Total bytes written : 0 > > Spillable Memory Manager spill count : 0 > > Total bags proactively spilled: 0 > > Total records proactively spilled: 0 > > > Job DAG: > > null > > > > 2015-01-21 22:04:31,444 [main] INFO > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - Failed! > > 2015-01-21 22:04:31,449 [main] ERROR org.apache.pig.tools.grunt.Grunt - > ERROR 1066: Unable to open iterator for alias data > > 2015-01-21 22:04:31,449 [main] ERROR org.apache.pig.tools.grunt.Grunt - > org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to > open iterator for alias data > > at org.apache.pig.PigServer.openIterator(PigServer.java:935) > > at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:746) > > at > > org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:372) > > at > > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:230) > > at > > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205) > > at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:66) > > at org.apache.pig.Main.run(Main.java:558) > > at org.apache.pig.Main.main(Main.java:170) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:606) > > at org.apache.hadoop.util.RunJar.run(RunJar.java:221) > > at org.apache.hadoop.util.RunJar.main(RunJar.java:136) > > Caused by: java.io.IOException: Job terminated with anomalous status FAILED > > at org.apache.pig.PigServer.openIterator(PigServer.java:927) > > ... 13 more > > > Details also at logfile: /private/tmp/pig_1421906665386.log > > ᐧ > > On Wed, Jan 21, 2015 at 9:53 PM, Russell Jurney <[email protected]> > wrote: > > > Nope, still fails. I've used Pig for over 5 years. If this is fucking me > > up, it must be a total nightmare for our average user. Pig doesn't work > out > > of the box. > > > > grunt> data = LOAD '/Users/rjurney/Software/foo/data/gsa_feed.xml' USING > > org.apache.pig.piggybank.storage.XMLLoader('record') AS (doc:chararray); > > > > grunt> dump data > > > > 2015-01-21 21:51:20,159 [main] INFO > > org.apache.pig.tools.pigstats.ScriptState - Pig features used in the > > script: UNKNOWN > > > > 2015-01-21 21:51:20,170 [main] INFO > > org.apache.pig.data.SchemaTupleBackend - Key [pig.schematuple] was not > > set... will not generate code. > > > > 2015-01-21 21:51:20,171 [main] INFO > > org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer - > > {RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, ConstantCalculator, > > GroupByConstParallelSetter, LimitOptimizer, LoadTypeCastInserter, > > MergeFilter, MergeForEach, PartitionFilterOptimizer, > > PredicatePushdownOptimizer, PushDownForEachFlatten, PushUpFilter, > > SplitFilter, StreamTypeCastInserter]} > > > > 2015-01-21 21:51:20,178 [main] INFO > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - > > File concatenation threshold: 100 optimistic? false > > > > 2015-01-21 21:51:20,180 [main] INFO > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer > > - MR plan size before optimization: 1 > > > > 2015-01-21 21:51:20,180 [main] INFO > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer > > - MR plan size after optimization: 1 > > > > 2015-01-21 21:51:20,193 [main] INFO > > org.apache.pig.tools.pigstats.mapreduce.MRScriptState - Pig script > settings > > are added to the job > > > > 2015-01-21 21:51:20,193 [main] INFO > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler > > - mapred.job.reduce.markreset.buffer.percent is not set, set to default > 0.3 > > > > 2015-01-21 21:51:20,201 [main] INFO > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler > > - Setting up single store job > > > > 2015-01-21 21:51:20,202 [main] INFO > > org.apache.pig.data.SchemaTupleFrontend - Key [pig.schematuple] is false, > > will not generate code. > > > > 2015-01-21 21:51:20,202 [main] INFO > > org.apache.pig.data.SchemaTupleFrontend - Starting process to move > > generated code to distributed cacche > > > > 2015-01-21 21:51:20,202 [main] INFO > > org.apache.pig.data.SchemaTupleFrontend - Distributed cache not supported > > or needed in local mode. Setting key [pig.schematuple.local.dir] with > code > > temp directory: > > /var/folders/0b/74l_65015_5fcbmbdz1w2xl40000gn/T/1421905880202-0 > > > > 2015-01-21 21:51:20,212 [main] INFO > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > > - 1 map-reduce job(s) waiting for submission. > > > > 2015-01-21 21:51:20,215 [JobControl] WARN > > org.apache.hadoop.mapred.JobClient - No job jar file set. User classes > may > > not be found. See JobConf(Class) or JobConf#setJar(String). > > > > 2015-01-21 21:51:20,238 [JobControl] INFO > > org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths > > to process : 1 > > > > 2015-01-21 21:51:20,238 [JobControl] INFO > > org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total > input > > paths to process : 1 > > > > 2015-01-21 21:51:20,238 [JobControl] INFO > > org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total > input > > paths (combined) to process : 6 > > > > 2015-01-21 21:51:20,320 [Thread-6] INFO org.apache.hadoop.mapred.Task - > > Using ResourceCalculatorPlugin : null > > > > 2015-01-21 21:51:20,330 [Thread-6] INFO > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader > > - Current split being processed > > file:/Users/rjurney/Software/foo/data/gsa_feed.xml:0+33554432 > > > > 2015-01-21 21:51:20,343 [Thread-6] WARN > > org.apache.hadoop.mapred.LocalJobRunner - job_local_0002 > > > > java.lang.IncompatibleClassChangeError: Found class > > org.apache.hadoop.mapreduce.TaskAttemptContext, but interface was > expected > > > > at > > > org.apache.pig.piggybank.storage.XMLLoader$XMLRecordReader.initialize(XMLLoader.java:102) > > > > at > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.initialize(PigRecordReader.java:181) > > > > at > > > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:522) > > > > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763) > > > > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) > > > > at > org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212) > > > > 2015-01-21 21:51:20,713 [main] INFO > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > > - HadoopJobId: job_local_0002 > > > > 2015-01-21 21:51:20,713 [main] INFO > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > > - Processing aliases data > > > > 2015-01-21 21:51:20,713 [main] INFO > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > > - detailed locations: M: data[2,7],data[-1,-1] C: R: > > > > 2015-01-21 21:51:20,715 [main] INFO > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > > - 0% complete > > > > 2015-01-21 21:51:20,717 [main] WARN > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > > - Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to > > stop immediately on failure. > > > > 2015-01-21 21:51:20,717 [main] INFO > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > > - job job_local_0002 has failed! Stop running all dependent jobs > > > > 2015-01-21 21:51:20,717 [main] INFO > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > > - 100% complete > > > > 2015-01-21 21:51:20,717 [main] ERROR > > org.apache.pig.tools.pigstats.mapreduce.MRPigStatsUtil - 1 map reduce > > job(s) failed! > > > > 2015-01-21 21:51:20,717 [main] INFO > > org.apache.pig.tools.pigstats.mapreduce.SimplePigStats - Script > Statistics: > > > > > > HadoopVersion PigVersion UserId StartedAt FinishedAt Features > > > > 1.0.4 0.14.0 rjurney 2015-01-21 21:51:20 2015-01-21 21:51:20 UNKNOWN > > > > > > Failed! > > > > > > Failed Jobs: > > > > JobId Alias Feature Message Outputs > > > > job_local_0002 data MAP_ONLY Message: Job failed! Error - NA > > file:/tmp/temp-476431088/tmp-2144425957, > > > > > > Input(s): > > > > Failed to read data from "/Users/rjurney/Software/foo/data/gsa_feed.xml" > > > > > > Output(s): > > > > Failed to produce result in "file:/tmp/temp-476431088/tmp-2144425957" > > > > > > Counters: > > > > Total records written : 0 > > > > Total bytes written : 0 > > > > Spillable Memory Manager spill count : 0 > > > > Total bags proactively spilled: 0 > > > > Total records proactively spilled: 0 > > > > > > Job DAG: > > > > job_local_0002 > > > > > > > > 2015-01-21 21:51:20,717 [main] INFO > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > > - Failed! > > > > 2015-01-21 21:51:20,718 [main] ERROR org.apache.pig.tools.grunt.Grunt - > > ERROR 1066: Unable to open iterator for alias data > > > > 2015-01-21 21:51:20,718 [main] ERROR org.apache.pig.tools.grunt.Grunt - > > org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to > > open iterator for alias data > > > > at org.apache.pig.PigServer.openIterator(PigServer.java:935) > > > > at > org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:746) > > > > at > > > org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:372) > > > > at > > > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:230) > > > > at > > > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205) > > > > at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:66) > > > > at org.apache.pig.Main.run(Main.java:558) > > > > at org.apache.pig.Main.main(Main.java:170) > > > > Caused by: java.io.IOException: Job terminated with anomalous status > FAILED > > > > at org.apache.pig.PigServer.openIterator(PigServer.java:927) > > > > ... 7 more > > > > > > Details also at logfile: /private/tmp/pig_1421905809654.log > > > > ᐧ > > > > On Wed, Jan 21, 2015 at 9:47 PM, Russell Jurney < > [email protected]> > > wrote: > > > >> To answer my own question, a typical YARN disaster. > >> > http://stackoverflow.com/questions/25236766/pig-error-unhandled-internal-error-found-interface-org-apache-hadoop-mapreduc > >> ᐧ > >> > >> On Wed, Jan 21, 2015 at 9:44 PM, Russell Jurney < > [email protected] > >> > wrote: > >> > >>> Wait... keep in mind there is no Hadoop on my system. How can I be > >>> getting Hadoop 1/2 issues? I can't use XMLLoader... > >>> > >>> grunt> REGISTER > >>> /Users/rjurney/Software/pig-0.14.0/contrib/piggybank/java/piggybank.jar > >>> > >>> grunt> > >>> > >>> grunt> data = LOAD '/Users/rjurney/Software/foo/data/gsa_feed.xml' > USING > >>> org.apache.pig.piggybank.storage.XMLLoader('record') AS > (doc:chararray); > >>> > >>> grunt> a = limit data 10; > >>> > >>> grunt> dump a > >>> > >>> 2015-01-21 21:42:52,113 [main] INFO > >>> org.apache.pig.tools.pigstats.ScriptState - Pig features used in the > >>> script: LIMIT > >>> > >>> 2015-01-21 21:42:52,125 [main] WARN > >>> org.apache.pig.data.SchemaTupleBackend - SchemaTupleBackend has already > >>> been initialized > >>> > >>> 2015-01-21 21:42:52,126 [main] INFO > >>> org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer - > >>> {RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, ConstantCalculator, > >>> GroupByConstParallelSetter, LimitOptimizer, LoadTypeCastInserter, > >>> MergeFilter, MergeForEach, PartitionFilterOptimizer, > >>> PredicatePushdownOptimizer, PushDownForEachFlatten, PushUpFilter, > >>> SplitFilter, StreamTypeCastInserter]} > >>> > >>> 2015-01-21 21:42:52,200 [main] INFO > >>> org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input > paths > >>> to process : 1 > >>> > >>> 2015-01-21 21:42:52,200 [main] INFO > >>> org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total > input > >>> paths to process : 1 > >>> > >>> 2015-01-21 21:42:52,204 [main] ERROR org.apache.pig.tools.grunt.Grunt - > >>> ERROR 2998: Unhandled internal error. Found class > >>> org.apache.hadoop.mapreduce.TaskAttemptContext, but interface was > expected > >>> > >>> 2015-01-21 21:42:52,204 [main] ERROR org.apache.pig.tools.grunt.Grunt - > >>> java.lang.IncompatibleClassChangeError: Found class > >>> org.apache.hadoop.mapreduce.TaskAttemptContext, but interface was > expected > >>> > >>> at > >>> > org.apache.pig.piggybank.storage.XMLLoader$XMLRecordReader.initialize(XMLLoader.java:102) > >>> > >>> at > >>> > org.apache.pig.impl.io.ReadToEndLoader.initializeReader(ReadToEndLoader.java:210) > >>> > >>> at > >>> > org.apache.pig.impl.io.ReadToEndLoader.getNextHelper(ReadToEndLoader.java:248) > >>> > >>> at > >>> > org.apache.pig.impl.io.ReadToEndLoader.getNext(ReadToEndLoader.java:229) > >>> > >>> at > >>> > org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLoad.getNextTuple(POLoad.java:137) > >>> > >>> at > >>> > org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:307) > >>> > >>> at > >>> > org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLimit.getNextTuple(POLimit.java:122) > >>> > >>> at > >>> > org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:307) > >>> > >>> at > >>> > org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNextTuple(POForEach.java:246) > >>> > >>> at > >>> > org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:307) > >>> > >>> at > >>> > org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStore.getNextTuple(POStore.java:159) > >>> > >>> at > >>> > org.apache.pig.backend.hadoop.executionengine.fetch.FetchLauncher.runPipeline(FetchLauncher.java:161) > >>> > >>> at > >>> > org.apache.pig.backend.hadoop.executionengine.fetch.FetchLauncher.launchPig(FetchLauncher.java:81) > >>> > >>> at > >>> > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:278) > >>> > >>> at org.apache.pig.PigServer.launchPlan(PigServer.java:1390) > >>> > >>> at > >>> > org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1375) > >>> > >>> at org.apache.pig.PigServer.storeEx(PigServer.java:1034) > >>> > >>> at org.apache.pig.PigServer.store(PigServer.java:997) > >>> > >>> at org.apache.pig.PigServer.openIterator(PigServer.java:910) > >>> > >>> at > >>> > org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:746) > >>> > >>> at > >>> > org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:372) > >>> > >>> at > >>> > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:230) > >>> > >>> at > >>> > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205) > >>> > >>> at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:66) > >>> > >>> at org.apache.pig.Main.run(Main.java:558) > >>> > >>> at org.apache.pig.Main.main(Main.java:170) > >>> > >>> > >>> Details also at logfile: /private/tmp/pig_1421905319118.log > >>> > >>> > >>> ᐧ > >>> > >>> On Wed, Jan 21, 2015 at 9:42 PM, Russell Jurney < > >>> [email protected]> wrote: > >>> > >>>> Not sure what was going on, but I got it working. > >>>> ᐧ > >>>> > >>>> On Wed, Jan 21, 2015 at 9:39 PM, Russell Jurney < > >>>> [email protected]> wrote: > >>>> > >>>>> I am working on a macbook without Hadoop installed. I download Pig > >>>>> 0.14.0, and I run it... > >>>>> > >>>>> Russells-MacBook-Pro:pig-0.14.0 rjurney$ bin/pig -l /tmp -v -w -x > local > >>>>> > >>>>> > >>>>> I run the following commands and get this exception. What gives? Why > >>>>> doesn't pig work? > >>>>> > >>>>> grunt> foo = LOAD '/etc/passwd' USING TextLoader(); > >>>>> > >>>>> 2015-01-21 21:36:24,095 [main] INFO > >>>>> org.apache.hadoop.conf.Configuration.deprecation - fs.default.name > is > >>>>> deprecated. Instead, use fs.defaultFS > >>>>> > >>>>> 2015-01-21 21:36:24,096 [main] INFO > >>>>> org.apache.hadoop.conf.Configuration.deprecation - > io.bytes.per.checksum is > >>>>> deprecated. Instead, use dfs.bytes-per-checksum > >>>>> > >>>>> grunt> dump foo > >>>>> > >>>>> 2015-01-21 21:36:25,701 [main] INFO > >>>>> org.apache.pig.tools.pigstats.ScriptState - Pig features used in the > >>>>> script: UNKNOWN > >>>>> > >>>>> 2015-01-21 21:36:25,735 [main] INFO > >>>>> org.apache.hadoop.conf.Configuration.deprecation - > io.bytes.per.checksum is > >>>>> deprecated. Instead, use dfs.bytes-per-checksum > >>>>> > >>>>> 2015-01-21 21:36:25,737 [main] INFO > >>>>> org.apache.hadoop.conf.Configuration.deprecation - fs.default.name > is > >>>>> deprecated. Instead, use fs.defaultFS > >>>>> > >>>>> 2015-01-21 21:36:25,738 [main] INFO > >>>>> org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer - > >>>>> {RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, ConstantCalculator, > >>>>> GroupByConstParallelSetter, LimitOptimizer, LoadTypeCastInserter, > >>>>> MergeFilter, MergeForEach, PartitionFilterOptimizer, > >>>>> PredicatePushdownOptimizer, PushDownForEachFlatten, PushUpFilter, > >>>>> SplitFilter, StreamTypeCastInserter]} > >>>>> > >>>>> 2015-01-21 21:36:25,740 [main] INFO > >>>>> > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - > >>>>> File concatenation threshold: 100 optimistic? false > >>>>> > >>>>> 2015-01-21 21:36:25,743 [main] INFO > >>>>> > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer > >>>>> - MR plan size before optimization: 1 > >>>>> > >>>>> 2015-01-21 21:36:25,743 [main] INFO > >>>>> > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer > >>>>> - MR plan size after optimization: 1 > >>>>> > >>>>> 2015-01-21 21:36:25,776 [main] INFO > >>>>> org.apache.hadoop.conf.Configuration.deprecation - > io.bytes.per.checksum is > >>>>> deprecated. Instead, use dfs.bytes-per-checksum > >>>>> > >>>>> 2015-01-21 21:36:25,777 [main] INFO > >>>>> org.apache.hadoop.conf.Configuration.deprecation - fs.default.name > is > >>>>> deprecated. Instead, use fs.defaultFS > >>>>> > >>>>> 2015-01-21 21:36:25,779 [main] ERROR org.apache.pig.tools.grunt.Grunt > >>>>> - ERROR 2998: Unhandled internal error. Bad type on operand stack > >>>>> > >>>>> Exception Details: > >>>>> > >>>>> Location: > >>>>> > >>>>> > >>>>> > org/apache/hadoop/mapred/JobTrackerInstrumentation.create(Lorg/apache/hadoop/mapred/JobTracker;Lorg/apache/hadoop/mapred/JobConf;)Lorg/apache/hadoop/mapred/JobTrackerInstrumentation; > >>>>> @5: invokestatic > >>>>> > >>>>> Reason: > >>>>> > >>>>> Type 'org/apache/hadoop/metrics2/lib/DefaultMetricsSystem' > >>>>> (current frame, stack[2]) is not assignable to > >>>>> 'org/apache/hadoop/metrics2/MetricsSystem' > >>>>> > >>>>> Current Frame: > >>>>> > >>>>> bci: @5 > >>>>> > >>>>> flags: { } > >>>>> > >>>>> locals: { 'org/apache/hadoop/mapred/JobTracker', > >>>>> 'org/apache/hadoop/mapred/JobConf' } > >>>>> > >>>>> stack: { 'org/apache/hadoop/mapred/JobTracker', > >>>>> 'org/apache/hadoop/mapred/JobConf', > >>>>> 'org/apache/hadoop/metrics2/lib/DefaultMetricsSystem' } > >>>>> > >>>>> Bytecode: > >>>>> > >>>>> 0000000: 2a2b b200 03b8 0004 b0 > >>>>> > >>>>> > >>>>> 2015-01-21 21:36:25,780 [main] ERROR org.apache.pig.tools.grunt.Grunt > >>>>> - java.lang.VerifyError: Bad type on operand stack > >>>>> > >>>>> Exception Details: > >>>>> > >>>>> Location: > >>>>> > >>>>> > >>>>> > org/apache/hadoop/mapred/JobTrackerInstrumentation.create(Lorg/apache/hadoop/mapred/JobTracker;Lorg/apache/hadoop/mapred/JobConf;)Lorg/apache/hadoop/mapred/JobTrackerInstrumentation; > >>>>> @5: invokestatic > >>>>> > >>>>> Reason: > >>>>> > >>>>> Type 'org/apache/hadoop/metrics2/lib/DefaultMetricsSystem' > >>>>> (current frame, stack[2]) is not assignable to > >>>>> 'org/apache/hadoop/metrics2/MetricsSystem' > >>>>> > >>>>> Current Frame: > >>>>> > >>>>> bci: @5 > >>>>> > >>>>> flags: { } > >>>>> > >>>>> locals: { 'org/apache/hadoop/mapred/JobTracker', > >>>>> 'org/apache/hadoop/mapred/JobConf' } > >>>>> > >>>>> stack: { 'org/apache/hadoop/mapred/JobTracker', > >>>>> 'org/apache/hadoop/mapred/JobConf', > >>>>> 'org/apache/hadoop/metrics2/lib/DefaultMetricsSystem' } > >>>>> > >>>>> Bytecode: > >>>>> > >>>>> 0000000: 2a2b b200 03b8 0004 b0 > >>>>> > >>>>> > >>>>> at > >>>>> > org.apache.hadoop.mapred.LocalJobRunner.<init>(LocalJobRunner.java:420) > >>>>> > >>>>> at org.apache.hadoop.mapred.JobClient.init(JobClient.java:472) > >>>>> > >>>>> at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:457) > >>>>> > >>>>> at > >>>>> > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:163) > >>>>> > >>>>> at > >>>>> > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:280) > >>>>> > >>>>> at org.apache.pig.PigServer.launchPlan(PigServer.java:1390) > >>>>> > >>>>> at > >>>>> > org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1375) > >>>>> > >>>>> at org.apache.pig.PigServer.storeEx(PigServer.java:1034) > >>>>> > >>>>> at org.apache.pig.PigServer.store(PigServer.java:997) > >>>>> > >>>>> at org.apache.pig.PigServer.openIterator(PigServer.java:910) > >>>>> > >>>>> at > >>>>> > org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:746) > >>>>> > >>>>> at > >>>>> > org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:372) > >>>>> > >>>>> at > >>>>> > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:230) > >>>>> > >>>>> at > >>>>> > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205) > >>>>> > >>>>> at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:66) > >>>>> > >>>>> at org.apache.pig.Main.run(Main.java:558) > >>>>> > >>>>> at org.apache.pig.Main.main(Main.java:170) > >>>>> > >>>>> > >>>>> Details also at logfile: /private/tmp/pig_1421904926350.log > >>>>> > >>>>> -- > >>>>> Russell Jurney twitter.com/rjurney [email protected] > >>>>> datasyndrome.com > >>>>> ᐧ > >>>>> > >>>> > >>>> > >>>> > >>>> -- > >>>> Russell Jurney twitter.com/rjurney [email protected] > >>>> datasyndrome.com > >>>> > >>> > >>> > >>> > >>> -- > >>> Russell Jurney twitter.com/rjurney [email protected] > datasyndrome > >>> .com > >>> > >> > >> > >> > >> -- > >> Russell Jurney twitter.com/rjurney [email protected] > datasyndrome. > >> com > >> > > > > > > > > -- > > Russell Jurney twitter.com/rjurney [email protected] > datasyndrome. > > com > > > > > > -- > Russell Jurney twitter.com/rjurney [email protected] > datasyndrome.com >
