Start a new thread and post your logs and I'll try to help. ᐧ On Wed, Jan 21, 2015 at 10:37 PM, Venkat Ramakrishnan < [email protected]> wrote:
> Hi Russell, > > I'm a newbie to Pig, and I was instructed to downgrade to Pig 0.13 > as the Pig 0.14 has issues with respect to jar files. I am on Windows 7. I > did > so, but still I face exceptions while running Pig 0.13. > > You have said that you have been working with Pig for 5 years. Is > it possible for you to take a look at my logs (Pig 0.13) and tell me > what am I doing wrong? If you can, I will send you the logs. > > Thanks in advance, > Venkat. > > > On Thu, Jan 22, 2015 at 11:36 AM, Russell Jurney <[email protected] > > > wrote: > > > Now I installed Hadoop 2.6, and I compiled both Pig and piggybank using > > 'ant -Dhadoopversion=23 jar' > > > > Still doesn't work... > > > > > > Russells-MacBook-Pro-OLD:java rjurney$ pig -l /tmp -v -w -x local > > > > 2015-01-21 22:04:25.808 java[34529:1903] Unable to load realm info from > > SCDynamicStore > > > > 15/01/21 22:04:26 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL > > > > 15/01/21 22:04:26 INFO pig.ExecTypeProvider: Picked LOCAL as the ExecType > > > > 2015-01-21 22:04:26,120 [main] INFO org.apache.pig.Main - Apache Pig > > version 0.14.0-SNAPSHOT (rUnversioned directory) compiled Jan 21 2015, > > 22:01:34 > > > > 2015-01-21 22:04:26,120 [main] INFO org.apache.pig.Main - Logging error > > messages to: /private/tmp/pig_1421906665386.log > > > > 2015-01-21 22:04:26,335 [main] INFO > > org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is > > deprecated. Instead, use fs.defaultFS > > > > 2015-01-21 22:04:26,335 [main] INFO > > org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is > > deprecated. Instead, use mapreduce.jobtracker.address > > > > 2015-01-21 22:04:26,337 [main] INFO > > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - > Connecting > > to hadoop file system at: file:/// > > > > 2015-01-21 22:04:26,439 [main] INFO > > org.apache.hadoop.conf.Configuration.deprecation - io.bytes.per.checksum > is > > deprecated. Instead, use dfs.bytes-per-checksum > > > > grunt> REGISTER > > /Users/rjurney/Software/pig-0.14.0/contrib/piggybank/java/piggybank.jar > > > > 2015-01-21 22:04:28,665 [main] INFO > > org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is > > deprecated. Instead, use fs.defaultFS > > > > 2015-01-21 22:04:28,668 [main] INFO > > org.apache.hadoop.conf.Configuration.deprecation - io.bytes.per.checksum > is > > deprecated. Instead, use dfs.bytes-per-checksum > > > > grunt> > > > > grunt> data = LOAD > '/Users/rjurney/Software/atomic_nile/data/gsa_feed.xml' > > USING org.apache.pig.piggybank.storage.XMLLoader('record') AS > > (doc:chararray); > > > > 2015-01-21 22:04:29,652 [main] INFO > > org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is > > deprecated. Instead, use fs.defaultFS > > > > 2015-01-21 22:04:29,654 [main] INFO > > org.apache.hadoop.conf.Configuration.deprecation - io.bytes.per.checksum > is > > deprecated. Instead, use dfs.bytes-per-checksum > > > > grunt> dump data > > > > 2015-01-21 22:04:30,940 [main] WARN > > org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop > > library for your platform... using builtin-java classes where applicable > > > > 2015-01-21 22:04:30,989 [main] INFO > > org.apache.pig.tools.pigstats.ScriptState - Pig features used in the > > script: UNKNOWN > > > > 2015-01-21 22:04:31,040 [main] INFO > > org.apache.hadoop.conf.Configuration.deprecation - io.bytes.per.checksum > is > > deprecated. Instead, use dfs.bytes-per-checksum > > > > 2015-01-21 22:04:31,042 [main] INFO > > org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is > > deprecated. Instead, use fs.defaultFS > > > > 2015-01-21 22:04:31,087 [main] INFO > > org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer - > > {RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, ConstantCalculator, > > GroupByConstParallelSetter, LimitOptimizer, LoadTypeCastInserter, > > MergeFilter, MergeForEach, PartitionFilterOptimizer, > > PredicatePushdownOptimizer, PushDownForEachFlatten, PushUpFilter, > > SplitFilter, StreamTypeCastInserter]} > > > > 2015-01-21 22:04:31,209 [main] INFO > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - > > File concatenation threshold: 100 optimistic? false > > > > 2015-01-21 22:04:31,235 [main] INFO > > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer > > - MR plan size before optimization: 1 > > > > 2015-01-21 22:04:31,235 [main] INFO > > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer > > - MR plan size after optimization: 1 > > > > 2015-01-21 22:04:31,268 [main] INFO > > org.apache.hadoop.conf.Configuration.deprecation - io.bytes.per.checksum > is > > deprecated. Instead, use dfs.bytes-per-checksum > > > > 2015-01-21 22:04:31,270 [main] INFO > > org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is > > deprecated. Instead, use fs.defaultFS > > > > 2015-01-21 22:04:31,285 [main] INFO > > org.apache.hadoop.conf.Configuration.deprecation - session.id is > > deprecated. Instead, use dfs.metrics.session-id > > > > 2015-01-21 22:04:31,286 [main] INFO > > org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics with > > processName=JobTracker, sessionId= > > > > 2015-01-21 22:04:31,322 [main] WARN > > org.apache.pig.backend.hadoop20.PigJobControl - falling back to default > > JobControl (not using hadoop 0.20 ?) > > > > java.lang.NoSuchFieldException: runnerState > > > > at java.lang.Class.getDeclaredField(Class.java:1938) > > > > at > > > > > org.apache.pig.backend.hadoop20.PigJobControl.<clinit>(PigJobControl.java:51) > > > > at > > > > > org.apache.pig.backend.hadoop.executionengine.shims.HadoopShims.newJobControl(HadoopShims.java:109) > > > > at > > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:314) > > > > at > > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:196) > > > > at > > > > > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:280) > > > > at org.apache.pig.PigServer.launchPlan(PigServer.java:1390) > > > > at > org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1375) > > > > at org.apache.pig.PigServer.storeEx(PigServer.java:1034) > > > > at org.apache.pig.PigServer.store(PigServer.java:997) > > > > at org.apache.pig.PigServer.openIterator(PigServer.java:910) > > > > at > org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:746) > > > > at > > > > > org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:372) > > > > at > > > > > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:230) > > > > at > > > > > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205) > > > > at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:66) > > > > at org.apache.pig.Main.run(Main.java:558) > > > > at org.apache.pig.Main.main(Main.java:170) > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > > at > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > > > > at > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > > > at java.lang.reflect.Method.invoke(Method.java:606) > > > > at org.apache.hadoop.util.RunJar.run(RunJar.java:221) > > > > at org.apache.hadoop.util.RunJar.main(RunJar.java:136) > > > > 2015-01-21 22:04:31,326 [main] INFO > > org.apache.pig.tools.pigstats.mapreduce.MRScriptState - Pig script > settings > > are added to the job > > > > 2015-01-21 22:04:31,331 [main] INFO > > org.apache.hadoop.conf.Configuration.deprecation - > > mapred.job.reduce.markreset.buffer.percent is deprecated. Instead, use > > mapreduce.reduce.markreset.buffer.percent > > > > 2015-01-21 22:04:31,331 [main] INFO > > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler > > - mapred.job.reduce.markreset.buffer.percent is not set, set to default > 0.3 > > > > 2015-01-21 22:04:31,331 [main] INFO > > org.apache.hadoop.conf.Configuration.deprecation - mapred.output.compress > > is deprecated. Instead, use mapreduce.output.fileoutputformat.compress > > > > 2015-01-21 22:04:31,364 [main] INFO > > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler > > - Setting up single store job > > > > 2015-01-21 22:04:31,374 [main] INFO > > org.apache.pig.data.SchemaTupleFrontend - Key [pig.schematuple] is false, > > will not generate code. > > > > 2015-01-21 22:04:31,374 [main] INFO > > org.apache.pig.data.SchemaTupleFrontend - Starting process to move > > generated code to distributed cacche > > > > 2015-01-21 22:04:31,374 [main] INFO > > org.apache.pig.data.SchemaTupleFrontend - Distributed cache not supported > > or needed in local mode. Setting key [pig.schematuple.local.dir] with > code > > temp directory: > > /var/folders/0b/74l_65015_5fcbmbdz1w2xl40000gn/T/1421906671373-0 > > > > 2015-01-21 22:04:31,406 [main] INFO > > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > > - 1 map-reduce job(s) waiting for submission. > > > > 2015-01-21 22:04:31,407 [main] INFO > > org.apache.hadoop.conf.Configuration.deprecation - > > mapred.job.tracker.http.address is deprecated. Instead, use > > mapreduce.jobtracker.http.address > > > > 2015-01-21 22:04:31,417 [JobControl] INFO > > org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics > > with processName=JobTracker, sessionId= - already initialized > > > > 2015-01-21 22:04:31,428 [JobControl] ERROR > > org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl - Error while > trying > > to run jobs. > > > > java.lang.IncompatibleClassChangeError: Found interface > > org.apache.hadoop.mapreduce.JobContext, but class was expected > > > > at > > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setupUdfEnvAndStores(PigOutputFormat.java:235) > > > > at > > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.checkOutputSpecs(PigOutputFormat.java:183) > > > > at > > > org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:562) > > > > at > > > > > org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:432) > > > > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296) > > > > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1293) > > > > at java.security.AccessController.doPrivileged(Native Method) > > > > at javax.security.auth.Subject.doAs(Subject.java:415) > > > > at > > > > > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) > > > > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293) > > > > at > > > > > org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335) > > > > at > > > > > org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:240) > > > > at > > org.apache.pig.backend.hadoop20.PigJobControl.run(PigJobControl.java:121) > > > > at java.lang.Thread.run(Thread.java:724) > > > > at > > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:276) > > > > 2015-01-21 22:04:31,429 [main] INFO > > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > > - 0% complete > > > > 2015-01-21 22:04:31,435 [main] WARN > > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > > - Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to > > stop immediately on failure. > > > > 2015-01-21 22:04:31,435 [main] INFO > > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > > - job null has failed! Stop running all dependent jobs > > > > 2015-01-21 22:04:31,435 [main] INFO > > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > > - 100% complete > > > > 2015-01-21 22:04:31,442 [main] ERROR > > org.apache.pig.tools.pigstats.mapreduce.MRPigStatsUtil - 1 map reduce > > job(s) failed! > > > > 2015-01-21 22:04:31,443 [main] INFO > > org.apache.pig.tools.pigstats.mapreduce.SimplePigStats - Script > Statistics: > > > > > > HadoopVersion PigVersion UserId StartedAt FinishedAt Features > > > > 2.6.0 0.14.0-SNAPSHOT rjurney 2015-01-21 22:04:31 2015-01-21 22:04:31 > > UNKNOWN > > > > > > Failed! > > > > > > Failed Jobs: > > > > JobId Alias Feature Message Outputs > > > > N/A data MAP_ONLY Message: Unexpected System Error Occured: > > java.lang.IncompatibleClassChangeError: Found interface > > org.apache.hadoop.mapreduce.JobContext, but class was expected > > > > at > > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setupUdfEnvAndStores(PigOutputFormat.java:235) > > > > at > > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.checkOutputSpecs(PigOutputFormat.java:183) > > > > at > > > org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:562) > > > > at > > > > > org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:432) > > > > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296) > > > > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1293) > > > > at java.security.AccessController.doPrivileged(Native Method) > > > > at javax.security.auth.Subject.doAs(Subject.java:415) > > > > at > > > > > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) > > > > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293) > > > > at > > > > > org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335) > > > > at > > > > > org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:240) > > > > at > > org.apache.pig.backend.hadoop20.PigJobControl.run(PigJobControl.java:121) > > > > at java.lang.Thread.run(Thread.java:724) > > > > at > > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:276) > > > > file:/tmp/temp1277658924/tmp730730286, > > > > > > Input(s): > > > > Failed to read data from > > "/Users/rjurney/Software/atomic_nile/data/gsa_feed.xml" > > > > > > Output(s): > > > > Failed to produce result in "file:/tmp/temp1277658924/tmp730730286" > > > > > > Counters: > > > > Total records written : 0 > > > > Total bytes written : 0 > > > > Spillable Memory Manager spill count : 0 > > > > Total bags proactively spilled: 0 > > > > Total records proactively spilled: 0 > > > > > > Job DAG: > > > > null > > > > > > > > 2015-01-21 22:04:31,444 [main] INFO > > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > > - Failed! > > > > 2015-01-21 22:04:31,449 [main] ERROR org.apache.pig.tools.grunt.Grunt - > > ERROR 1066: Unable to open iterator for alias data > > > > 2015-01-21 22:04:31,449 [main] ERROR org.apache.pig.tools.grunt.Grunt - > > org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to > > open iterator for alias data > > > > at org.apache.pig.PigServer.openIterator(PigServer.java:935) > > > > at > org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:746) > > > > at > > > > > org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:372) > > > > at > > > > > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:230) > > > > at > > > > > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205) > > > > at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:66) > > > > at org.apache.pig.Main.run(Main.java:558) > > > > at org.apache.pig.Main.main(Main.java:170) > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > > at > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > > > > at > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > > > at java.lang.reflect.Method.invoke(Method.java:606) > > > > at org.apache.hadoop.util.RunJar.run(RunJar.java:221) > > > > at org.apache.hadoop.util.RunJar.main(RunJar.java:136) > > > > Caused by: java.io.IOException: Job terminated with anomalous status > FAILED > > > > at org.apache.pig.PigServer.openIterator(PigServer.java:927) > > > > ... 13 more > > > > > > Details also at logfile: /private/tmp/pig_1421906665386.log > > > > ᐧ > > > > On Wed, Jan 21, 2015 at 9:53 PM, Russell Jurney < > [email protected]> > > wrote: > > > > > Nope, still fails. I've used Pig for over 5 years. If this is fucking > me > > > up, it must be a total nightmare for our average user. Pig doesn't work > > out > > > of the box. > > > > > > grunt> data = LOAD '/Users/rjurney/Software/foo/data/gsa_feed.xml' > USING > > > org.apache.pig.piggybank.storage.XMLLoader('record') AS > (doc:chararray); > > > > > > grunt> dump data > > > > > > 2015-01-21 21:51:20,159 [main] INFO > > > org.apache.pig.tools.pigstats.ScriptState - Pig features used in the > > > script: UNKNOWN > > > > > > 2015-01-21 21:51:20,170 [main] INFO > > > org.apache.pig.data.SchemaTupleBackend - Key [pig.schematuple] was not > > > set... will not generate code. > > > > > > 2015-01-21 21:51:20,171 [main] INFO > > > org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer - > > > {RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, ConstantCalculator, > > > GroupByConstParallelSetter, LimitOptimizer, LoadTypeCastInserter, > > > MergeFilter, MergeForEach, PartitionFilterOptimizer, > > > PredicatePushdownOptimizer, PushDownForEachFlatten, PushUpFilter, > > > SplitFilter, StreamTypeCastInserter]} > > > > > > 2015-01-21 21:51:20,178 [main] INFO > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - > > > File concatenation threshold: 100 optimistic? false > > > > > > 2015-01-21 21:51:20,180 [main] INFO > > > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer > > > - MR plan size before optimization: 1 > > > > > > 2015-01-21 21:51:20,180 [main] INFO > > > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer > > > - MR plan size after optimization: 1 > > > > > > 2015-01-21 21:51:20,193 [main] INFO > > > org.apache.pig.tools.pigstats.mapreduce.MRScriptState - Pig script > > settings > > > are added to the job > > > > > > 2015-01-21 21:51:20,193 [main] INFO > > > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler > > > - mapred.job.reduce.markreset.buffer.percent is not set, set to default > > 0.3 > > > > > > 2015-01-21 21:51:20,201 [main] INFO > > > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler > > > - Setting up single store job > > > > > > 2015-01-21 21:51:20,202 [main] INFO > > > org.apache.pig.data.SchemaTupleFrontend - Key [pig.schematuple] is > false, > > > will not generate code. > > > > > > 2015-01-21 21:51:20,202 [main] INFO > > > org.apache.pig.data.SchemaTupleFrontend - Starting process to move > > > generated code to distributed cacche > > > > > > 2015-01-21 21:51:20,202 [main] INFO > > > org.apache.pig.data.SchemaTupleFrontend - Distributed cache not > supported > > > or needed in local mode. Setting key [pig.schematuple.local.dir] with > > code > > > temp directory: > > > /var/folders/0b/74l_65015_5fcbmbdz1w2xl40000gn/T/1421905880202-0 > > > > > > 2015-01-21 21:51:20,212 [main] INFO > > > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > > > - 1 map-reduce job(s) waiting for submission. > > > > > > 2015-01-21 21:51:20,215 [JobControl] WARN > > > org.apache.hadoop.mapred.JobClient - No job jar file set. User classes > > may > > > not be found. See JobConf(Class) or JobConf#setJar(String). > > > > > > 2015-01-21 21:51:20,238 [JobControl] INFO > > > org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input > paths > > > to process : 1 > > > > > > 2015-01-21 21:51:20,238 [JobControl] INFO > > > org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total > > input > > > paths to process : 1 > > > > > > 2015-01-21 21:51:20,238 [JobControl] INFO > > > org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total > > input > > > paths (combined) to process : 6 > > > > > > 2015-01-21 21:51:20,320 [Thread-6] INFO org.apache.hadoop.mapred.Task > - > > > Using ResourceCalculatorPlugin : null > > > > > > 2015-01-21 21:51:20,330 [Thread-6] INFO > > > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader > > > - Current split being processed > > > file:/Users/rjurney/Software/foo/data/gsa_feed.xml:0+33554432 > > > > > > 2015-01-21 21:51:20,343 [Thread-6] WARN > > > org.apache.hadoop.mapred.LocalJobRunner - job_local_0002 > > > > > > java.lang.IncompatibleClassChangeError: Found class > > > org.apache.hadoop.mapreduce.TaskAttemptContext, but interface was > > expected > > > > > > at > > > > > > org.apache.pig.piggybank.storage.XMLLoader$XMLRecordReader.initialize(XMLLoader.java:102) > > > > > > at > > > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.initialize(PigRecordReader.java:181) > > > > > > at > > > > > > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:522) > > > > > > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763) > > > > > > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) > > > > > > at > > org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212) > > > > > > 2015-01-21 21:51:20,713 [main] INFO > > > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > > > - HadoopJobId: job_local_0002 > > > > > > 2015-01-21 21:51:20,713 [main] INFO > > > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > > > - Processing aliases data > > > > > > 2015-01-21 21:51:20,713 [main] INFO > > > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > > > - detailed locations: M: data[2,7],data[-1,-1] C: R: > > > > > > 2015-01-21 21:51:20,715 [main] INFO > > > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > > > - 0% complete > > > > > > 2015-01-21 21:51:20,717 [main] WARN > > > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > > > - Ooops! Some job has failed! Specify -stop_on_failure if you want Pig > to > > > stop immediately on failure. > > > > > > 2015-01-21 21:51:20,717 [main] INFO > > > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > > > - job job_local_0002 has failed! Stop running all dependent jobs > > > > > > 2015-01-21 21:51:20,717 [main] INFO > > > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > > > - 100% complete > > > > > > 2015-01-21 21:51:20,717 [main] ERROR > > > org.apache.pig.tools.pigstats.mapreduce.MRPigStatsUtil - 1 map reduce > > > job(s) failed! > > > > > > 2015-01-21 21:51:20,717 [main] INFO > > > org.apache.pig.tools.pigstats.mapreduce.SimplePigStats - Script > > Statistics: > > > > > > > > > HadoopVersion PigVersion UserId StartedAt FinishedAt Features > > > > > > 1.0.4 0.14.0 rjurney 2015-01-21 21:51:20 2015-01-21 21:51:20 UNKNOWN > > > > > > > > > Failed! > > > > > > > > > Failed Jobs: > > > > > > JobId Alias Feature Message Outputs > > > > > > job_local_0002 data MAP_ONLY Message: Job failed! Error - NA > > > file:/tmp/temp-476431088/tmp-2144425957, > > > > > > > > > Input(s): > > > > > > Failed to read data from > "/Users/rjurney/Software/foo/data/gsa_feed.xml" > > > > > > > > > Output(s): > > > > > > Failed to produce result in "file:/tmp/temp-476431088/tmp-2144425957" > > > > > > > > > Counters: > > > > > > Total records written : 0 > > > > > > Total bytes written : 0 > > > > > > Spillable Memory Manager spill count : 0 > > > > > > Total bags proactively spilled: 0 > > > > > > Total records proactively spilled: 0 > > > > > > > > > Job DAG: > > > > > > job_local_0002 > > > > > > > > > > > > 2015-01-21 21:51:20,717 [main] INFO > > > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > > > - Failed! > > > > > > 2015-01-21 21:51:20,718 [main] ERROR org.apache.pig.tools.grunt.Grunt - > > > ERROR 1066: Unable to open iterator for alias data > > > > > > 2015-01-21 21:51:20,718 [main] ERROR org.apache.pig.tools.grunt.Grunt - > > > org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable > to > > > open iterator for alias data > > > > > > at org.apache.pig.PigServer.openIterator(PigServer.java:935) > > > > > > at > > org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:746) > > > > > > at > > > > > > org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:372) > > > > > > at > > > > > > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:230) > > > > > > at > > > > > > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205) > > > > > > at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:66) > > > > > > at org.apache.pig.Main.run(Main.java:558) > > > > > > at org.apache.pig.Main.main(Main.java:170) > > > > > > Caused by: java.io.IOException: Job terminated with anomalous status > > FAILED > > > > > > at org.apache.pig.PigServer.openIterator(PigServer.java:927) > > > > > > ... 7 more > > > > > > > > > Details also at logfile: /private/tmp/pig_1421905809654.log > > > > > > ᐧ > > > > > > On Wed, Jan 21, 2015 at 9:47 PM, Russell Jurney < > > [email protected]> > > > wrote: > > > > > >> To answer my own question, a typical YARN disaster. > > >> > > > http://stackoverflow.com/questions/25236766/pig-error-unhandled-internal-error-found-interface-org-apache-hadoop-mapreduc > > >> ᐧ > > >> > > >> On Wed, Jan 21, 2015 at 9:44 PM, Russell Jurney < > > [email protected] > > >> > wrote: > > >> > > >>> Wait... keep in mind there is no Hadoop on my system. How can I be > > >>> getting Hadoop 1/2 issues? I can't use XMLLoader... > > >>> > > >>> grunt> REGISTER > > >>> > /Users/rjurney/Software/pig-0.14.0/contrib/piggybank/java/piggybank.jar > > >>> > > >>> grunt> > > >>> > > >>> grunt> data = LOAD '/Users/rjurney/Software/foo/data/gsa_feed.xml' > > USING > > >>> org.apache.pig.piggybank.storage.XMLLoader('record') AS > > (doc:chararray); > > >>> > > >>> grunt> a = limit data 10; > > >>> > > >>> grunt> dump a > > >>> > > >>> 2015-01-21 21:42:52,113 [main] INFO > > >>> org.apache.pig.tools.pigstats.ScriptState - Pig features used in the > > >>> script: LIMIT > > >>> > > >>> 2015-01-21 21:42:52,125 [main] WARN > > >>> org.apache.pig.data.SchemaTupleBackend - SchemaTupleBackend has > already > > >>> been initialized > > >>> > > >>> 2015-01-21 21:42:52,126 [main] INFO > > >>> org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer - > > >>> {RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, ConstantCalculator, > > >>> GroupByConstParallelSetter, LimitOptimizer, LoadTypeCastInserter, > > >>> MergeFilter, MergeForEach, PartitionFilterOptimizer, > > >>> PredicatePushdownOptimizer, PushDownForEachFlatten, PushUpFilter, > > >>> SplitFilter, StreamTypeCastInserter]} > > >>> > > >>> 2015-01-21 21:42:52,200 [main] INFO > > >>> org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input > > paths > > >>> to process : 1 > > >>> > > >>> 2015-01-21 21:42:52,200 [main] INFO > > >>> org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total > > input > > >>> paths to process : 1 > > >>> > > >>> 2015-01-21 21:42:52,204 [main] ERROR > org.apache.pig.tools.grunt.Grunt - > > >>> ERROR 2998: Unhandled internal error. Found class > > >>> org.apache.hadoop.mapreduce.TaskAttemptContext, but interface was > > expected > > >>> > > >>> 2015-01-21 21:42:52,204 [main] ERROR > org.apache.pig.tools.grunt.Grunt - > > >>> java.lang.IncompatibleClassChangeError: Found class > > >>> org.apache.hadoop.mapreduce.TaskAttemptContext, but interface was > > expected > > >>> > > >>> at > > >>> > > > org.apache.pig.piggybank.storage.XMLLoader$XMLRecordReader.initialize(XMLLoader.java:102) > > >>> > > >>> at > > >>> > > > org.apache.pig.impl.io.ReadToEndLoader.initializeReader(ReadToEndLoader.java:210) > > >>> > > >>> at > > >>> > > > org.apache.pig.impl.io.ReadToEndLoader.getNextHelper(ReadToEndLoader.java:248) > > >>> > > >>> at > > >>> > > org.apache.pig.impl.io.ReadToEndLoader.getNext(ReadToEndLoader.java:229) > > >>> > > >>> at > > >>> > > > org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLoad.getNextTuple(POLoad.java:137) > > >>> > > >>> at > > >>> > > > org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:307) > > >>> > > >>> at > > >>> > > > org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLimit.getNextTuple(POLimit.java:122) > > >>> > > >>> at > > >>> > > > org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:307) > > >>> > > >>> at > > >>> > > > org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNextTuple(POForEach.java:246) > > >>> > > >>> at > > >>> > > > org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:307) > > >>> > > >>> at > > >>> > > > org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStore.getNextTuple(POStore.java:159) > > >>> > > >>> at > > >>> > > > org.apache.pig.backend.hadoop.executionengine.fetch.FetchLauncher.runPipeline(FetchLauncher.java:161) > > >>> > > >>> at > > >>> > > > org.apache.pig.backend.hadoop.executionengine.fetch.FetchLauncher.launchPig(FetchLauncher.java:81) > > >>> > > >>> at > > >>> > > > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:278) > > >>> > > >>> at org.apache.pig.PigServer.launchPlan(PigServer.java:1390) > > >>> > > >>> at > > >>> > > org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1375) > > >>> > > >>> at org.apache.pig.PigServer.storeEx(PigServer.java:1034) > > >>> > > >>> at org.apache.pig.PigServer.store(PigServer.java:997) > > >>> > > >>> at org.apache.pig.PigServer.openIterator(PigServer.java:910) > > >>> > > >>> at > > >>> > > org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:746) > > >>> > > >>> at > > >>> > > > org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:372) > > >>> > > >>> at > > >>> > > > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:230) > > >>> > > >>> at > > >>> > > > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205) > > >>> > > >>> at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:66) > > >>> > > >>> at org.apache.pig.Main.run(Main.java:558) > > >>> > > >>> at org.apache.pig.Main.main(Main.java:170) > > >>> > > >>> > > >>> Details also at logfile: /private/tmp/pig_1421905319118.log > > >>> > > >>> > > >>> ᐧ > > >>> > > >>> On Wed, Jan 21, 2015 at 9:42 PM, Russell Jurney < > > >>> [email protected]> wrote: > > >>> > > >>>> Not sure what was going on, but I got it working. > > >>>> ᐧ > > >>>> > > >>>> On Wed, Jan 21, 2015 at 9:39 PM, Russell Jurney < > > >>>> [email protected]> wrote: > > >>>> > > >>>>> I am working on a macbook without Hadoop installed. I download Pig > > >>>>> 0.14.0, and I run it... > > >>>>> > > >>>>> Russells-MacBook-Pro:pig-0.14.0 rjurney$ bin/pig -l /tmp -v -w -x > > local > > >>>>> > > >>>>> > > >>>>> I run the following commands and get this exception. What gives? > Why > > >>>>> doesn't pig work? > > >>>>> > > >>>>> grunt> foo = LOAD '/etc/passwd' USING TextLoader(); > > >>>>> > > >>>>> 2015-01-21 21:36:24,095 [main] INFO > > >>>>> org.apache.hadoop.conf.Configuration.deprecation - fs.default.name > > is > > >>>>> deprecated. Instead, use fs.defaultFS > > >>>>> > > >>>>> 2015-01-21 21:36:24,096 [main] INFO > > >>>>> org.apache.hadoop.conf.Configuration.deprecation - > > io.bytes.per.checksum is > > >>>>> deprecated. Instead, use dfs.bytes-per-checksum > > >>>>> > > >>>>> grunt> dump foo > > >>>>> > > >>>>> 2015-01-21 21:36:25,701 [main] INFO > > >>>>> org.apache.pig.tools.pigstats.ScriptState - Pig features used in > the > > >>>>> script: UNKNOWN > > >>>>> > > >>>>> 2015-01-21 21:36:25,735 [main] INFO > > >>>>> org.apache.hadoop.conf.Configuration.deprecation - > > io.bytes.per.checksum is > > >>>>> deprecated. Instead, use dfs.bytes-per-checksum > > >>>>> > > >>>>> 2015-01-21 21:36:25,737 [main] INFO > > >>>>> org.apache.hadoop.conf.Configuration.deprecation - fs.default.name > > is > > >>>>> deprecated. Instead, use fs.defaultFS > > >>>>> > > >>>>> 2015-01-21 21:36:25,738 [main] INFO > > >>>>> org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer - > > >>>>> {RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, ConstantCalculator, > > >>>>> GroupByConstParallelSetter, LimitOptimizer, LoadTypeCastInserter, > > >>>>> MergeFilter, MergeForEach, PartitionFilterOptimizer, > > >>>>> PredicatePushdownOptimizer, PushDownForEachFlatten, PushUpFilter, > > >>>>> SplitFilter, StreamTypeCastInserter]} > > >>>>> > > >>>>> 2015-01-21 21:36:25,740 [main] INFO > > >>>>> > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - > > >>>>> File concatenation threshold: 100 optimistic? false > > >>>>> > > >>>>> 2015-01-21 21:36:25,743 [main] INFO > > >>>>> > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer > > >>>>> - MR plan size before optimization: 1 > > >>>>> > > >>>>> 2015-01-21 21:36:25,743 [main] INFO > > >>>>> > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer > > >>>>> - MR plan size after optimization: 1 > > >>>>> > > >>>>> 2015-01-21 21:36:25,776 [main] INFO > > >>>>> org.apache.hadoop.conf.Configuration.deprecation - > > io.bytes.per.checksum is > > >>>>> deprecated. Instead, use dfs.bytes-per-checksum > > >>>>> > > >>>>> 2015-01-21 21:36:25,777 [main] INFO > > >>>>> org.apache.hadoop.conf.Configuration.deprecation - fs.default.name > > is > > >>>>> deprecated. Instead, use fs.defaultFS > > >>>>> > > >>>>> 2015-01-21 21:36:25,779 [main] ERROR > org.apache.pig.tools.grunt.Grunt > > >>>>> - ERROR 2998: Unhandled internal error. Bad type on operand stack > > >>>>> > > >>>>> Exception Details: > > >>>>> > > >>>>> Location: > > >>>>> > > >>>>> > > >>>>> > > > org/apache/hadoop/mapred/JobTrackerInstrumentation.create(Lorg/apache/hadoop/mapred/JobTracker;Lorg/apache/hadoop/mapred/JobConf;)Lorg/apache/hadoop/mapred/JobTrackerInstrumentation; > > >>>>> @5: invokestatic > > >>>>> > > >>>>> Reason: > > >>>>> > > >>>>> Type 'org/apache/hadoop/metrics2/lib/DefaultMetricsSystem' > > >>>>> (current frame, stack[2]) is not assignable to > > >>>>> 'org/apache/hadoop/metrics2/MetricsSystem' > > >>>>> > > >>>>> Current Frame: > > >>>>> > > >>>>> bci: @5 > > >>>>> > > >>>>> flags: { } > > >>>>> > > >>>>> locals: { 'org/apache/hadoop/mapred/JobTracker', > > >>>>> 'org/apache/hadoop/mapred/JobConf' } > > >>>>> > > >>>>> stack: { 'org/apache/hadoop/mapred/JobTracker', > > >>>>> 'org/apache/hadoop/mapred/JobConf', > > >>>>> 'org/apache/hadoop/metrics2/lib/DefaultMetricsSystem' } > > >>>>> > > >>>>> Bytecode: > > >>>>> > > >>>>> 0000000: 2a2b b200 03b8 0004 b0 > > >>>>> > > >>>>> > > >>>>> 2015-01-21 21:36:25,780 [main] ERROR > org.apache.pig.tools.grunt.Grunt > > >>>>> - java.lang.VerifyError: Bad type on operand stack > > >>>>> > > >>>>> Exception Details: > > >>>>> > > >>>>> Location: > > >>>>> > > >>>>> > > >>>>> > > > org/apache/hadoop/mapred/JobTrackerInstrumentation.create(Lorg/apache/hadoop/mapred/JobTracker;Lorg/apache/hadoop/mapred/JobConf;)Lorg/apache/hadoop/mapred/JobTrackerInstrumentation; > > >>>>> @5: invokestatic > > >>>>> > > >>>>> Reason: > > >>>>> > > >>>>> Type 'org/apache/hadoop/metrics2/lib/DefaultMetricsSystem' > > >>>>> (current frame, stack[2]) is not assignable to > > >>>>> 'org/apache/hadoop/metrics2/MetricsSystem' > > >>>>> > > >>>>> Current Frame: > > >>>>> > > >>>>> bci: @5 > > >>>>> > > >>>>> flags: { } > > >>>>> > > >>>>> locals: { 'org/apache/hadoop/mapred/JobTracker', > > >>>>> 'org/apache/hadoop/mapred/JobConf' } > > >>>>> > > >>>>> stack: { 'org/apache/hadoop/mapred/JobTracker', > > >>>>> 'org/apache/hadoop/mapred/JobConf', > > >>>>> 'org/apache/hadoop/metrics2/lib/DefaultMetricsSystem' } > > >>>>> > > >>>>> Bytecode: > > >>>>> > > >>>>> 0000000: 2a2b b200 03b8 0004 b0 > > >>>>> > > >>>>> > > >>>>> at > > >>>>> > > org.apache.hadoop.mapred.LocalJobRunner.<init>(LocalJobRunner.java:420) > > >>>>> > > >>>>> at org.apache.hadoop.mapred.JobClient.init(JobClient.java:472) > > >>>>> > > >>>>> at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:457) > > >>>>> > > >>>>> at > > >>>>> > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:163) > > >>>>> > > >>>>> at > > >>>>> > > > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:280) > > >>>>> > > >>>>> at org.apache.pig.PigServer.launchPlan(PigServer.java:1390) > > >>>>> > > >>>>> at > > >>>>> > > org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1375) > > >>>>> > > >>>>> at org.apache.pig.PigServer.storeEx(PigServer.java:1034) > > >>>>> > > >>>>> at org.apache.pig.PigServer.store(PigServer.java:997) > > >>>>> > > >>>>> at org.apache.pig.PigServer.openIterator(PigServer.java:910) > > >>>>> > > >>>>> at > > >>>>> > > org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:746) > > >>>>> > > >>>>> at > > >>>>> > > > org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:372) > > >>>>> > > >>>>> at > > >>>>> > > > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:230) > > >>>>> > > >>>>> at > > >>>>> > > > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205) > > >>>>> > > >>>>> at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:66) > > >>>>> > > >>>>> at org.apache.pig.Main.run(Main.java:558) > > >>>>> > > >>>>> at org.apache.pig.Main.main(Main.java:170) > > >>>>> > > >>>>> > > >>>>> Details also at logfile: /private/tmp/pig_1421904926350.log > > >>>>> > > >>>>> -- > > >>>>> Russell Jurney twitter.com/rjurney [email protected] > > >>>>> datasyndrome.com > > >>>>> ᐧ > > >>>>> > > >>>> > > >>>> > > >>>> > > >>>> -- > > >>>> Russell Jurney twitter.com/rjurney [email protected] > > >>>> datasyndrome.com > > >>>> > > >>> > > >>> > > >>> > > >>> -- > > >>> Russell Jurney twitter.com/rjurney [email protected] > > datasyndrome > > >>> .com > > >>> > > >> > > >> > > >> > > >> -- > > >> Russell Jurney twitter.com/rjurney [email protected] > > datasyndrome. > > >> com > > >> > > > > > > > > > > > > -- > > > Russell Jurney twitter.com/rjurney [email protected] > > datasyndrome. > > > com > > > > > > > > > > > -- > > Russell Jurney twitter.com/rjurney [email protected] > > datasyndrome.com > > > -- Russell Jurney twitter.com/rjurney [email protected] datasyndrome.com
