This blog post describes some of this behavior http://hadoopified.wordpress.com/2013/04/07/pig-startup-script-behavior/
On Mon, May 6, 2013 at 1:07 AM, <[email protected]> wrote: > forget it ! It came from hadoop-env.sh provided by the package I installed > on one cluster :( > > Regards > -- > Cyril SCETBON > > On May 3, 2013, at 11:44 AM, [email protected]<mailto: > [email protected]> wrote: > > Hi, > > I have 2 differents behaviour for the same Pig version (with hadoop 1.1.2) > on different servers. If anyone can tell me why with the same versions and > the same parameters I don't have the same behaviour. I'm trying to set the > builtin import list but it's not used with one hadoop cluster whereas it's > used when I use it on a single node cluster : > > + PIG_OPTS=' -Dudf.import.list=org.apache.cassandra.hadoop.pig > -Dpig.additional.jars=/usr/local/Cellar/cassandra/1.2.4/apache-cassandra-thrift-1.2.4.jar:/usr/local/Cellar/cassandra/1.2.4/libthrift-0.7.0.jar > -Dpig.log.dir=/usr/local/bin/../Cellar/pig/0.11.1/bin/../logs' > + PIG_OPTS=' -Dudf.import.list=org.apache.cassandra.hadoop.pig > -Dpig.additional.jars=/usr/local/Cellar/cassandra/1.2.4/apache-cassandra-thrift-1.2.4.jar:/usr/local/Cellar/cassandra/1.2.4/libthrift-0.7.0.jar > -Dpig.log.dir=/usr/local/bin/../Cellar/pig/0.11.1/bin/../logs > -Dpig.log.file=pig.log' > + PIG_OPTS=' -Dudf.import.list=org.apache.cassandra.hadoop.pig > -Dpig.additional.jars=/usr/local/Cellar/cassandra/1.2.4/apache-cassandra-thrift-1.2.4.jar:/usr/local/Cellar/cassandra/1.2.4/libthrift-0.7.0.jar > -Dpig.log.dir=/usr/local/bin/../Cellar/pig/0.11.1/bin/../logs > -Dpig.log.file=pig.log > -Dpig.home.dir=/usr/local/bin/../Cellar/pig/0.11.1/bin/..' > + '[' '' == true ']' > + '[' -n /usr/local/bin/hadoop ']' > + '[' false == true ']' > + '[' -f /usr/local/bin/../Cellar/pig/0.11.1/bin/../pig-withouthadoop.jar > ']' > ++ echo > /usr/local/bin/../Cellar/pig/0.11.1/bin/../pig-0.11.1-withouthadoop.jar > + > PIG_JAR=/usr/local/bin/../Cellar/pig/0.11.1/bin/../pig-0.11.1-withouthadoop.jar > + '[' -z > /usr/local/bin/../Cellar/pig/0.11.1/bin/../pig-0.11.1-withouthadoop.jar ']' > + '[' -n > /usr/local/bin/../Cellar/pig/0.11.1/bin/../pig-0.11.1-withouthadoop.jar ']' > + > CLASSPATH='/etc/hadoop/conf:/usr/lib/jvm/java-7-oracle/lib/tools.jar:::/opt/apache-cassandra-1.2.3/lib/*:/usr/share/hadoop/lib/*:/opt/pig-0.11.1/pig-0.11.1-withouthadoop.jar:/etc/hadoop:/opt/pig-0.11.1/bin/../lib/jython-standalone-2.5.3.jar:/opt/pig-0.11.1/bin/../pig-0.11.1-withouthadoop.jar' > + export > 'HADOOP_CLASSPATH=/etc/hadoop/conf:/usr/lib/jvm/java-7-oracle/lib/tools.jar:::/opt/apache-cassandra-1.2.3/lib/*:/usr/share/hadoop/lib/*:/opt/pig-0.11.1/pig-0.11.1-withouthadoop.jar:/etc/hadoop:/opt/pig-0.11.1/bin/../lib/jython-standalone-2.5.3.jar:/opt/pig-0.11.1/bin/../pig-0.11.1-withouthadoop.jar:' > + > HADOOP_CLASSPATH='/etc/hadoop/conf:/usr/lib/jvm/java-7-oracle/lib/tools.jar:::/opt/apache-cassandra-1.2.3/lib/*:/usr/share/hadoop/lib/*:/opt/pig-0.11.1/pig-0.11.1-withouthadoop.jar:/etc/hadoop:/opt/pig-0.11.1/bin/../lib/jython-standalone-2.5.3.jar:/opt/pig-0.11.1/bin/../pig-0.11.1-withouthadoop.jar:' > + export 'HADOOP_OPTS=-Xmx1000m > -Dudf.import.list=org.apache.cassandra.hadoop.pig > -Dpig.additional.jars=/opt/apache-cassandra-1.2.3/lib/apache-cassandra-thrift-1.2.3.jar:/opt/apache-cassandra-1.2.3/lib/libthrift-0.7.0.jar > -4 log4j.properties -Dpig.log.dir=/opt/pig-0.11.1/bin/../logs > -Dpig.log.file=pig.log -Dpig.home.dir=/opt/pig-0.11.1/bin/.. > -Djava.net.preferIPv4Stack=true ' > + HADOOP_OPTS='-Xmx1000m > -Dudf.import.list=org.apache.cassandra.hadoop.pig > -Dpig.additional.jars=/opt/apache-cassandra-1.2.3/lib/apache-cassandra-thrift-1.2.3.jar:/opt/apache-cassandra-1.2.3/lib/libthrift-0.7.0.jar > -4 log4j.properties -Dpig.log.dir=/opt/pig-0.11.1/bin/../logs > -Dpig.log.file=pig.log -Dpig.home.dir=/opt/pig-0.11.1/bin/.. > -Djava.net.preferIPv4Stack=true ' > + '[' false == true ']' > + exec /usr/bin/hadoop jar > /opt/pig-0.11.1/bin/../pig-0.11.1-withouthadoop.jar ./hadoop-sum-value_1.pig > 2013-05-03 07:57:55,140 [main] INFO org.apache.pig.Main - Apache Pig > version 0.11.1 (r1459641) compiled Mar 22 2013, 02:13:53 > 2013-05-03 07:57:55,141 [main] INFO org.apache.pig.Main - Logging error > messages to: /home/ubuntu/pig_1367567875107.log > 2013-05-03 07:57:57,502 [main] INFO org.apache.pig.impl.util.Utils - > Default bootup file /home/ubuntu/.pigbootup not found > 2013-05-03 07:57:58,461 [main] INFO > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - > Connecting to hadoop file system at: hdfs://hadoop-cass-1:9100 > 2013-05-03 07:57:59,663 [main] INFO > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - > Connecting to map-reduce job tracker at: hadoop-cass-1:8021 > 2013-05-03 07:58:01,387 [main] ERROR org.apache.pig.tools.grunt.Grunt - > ERROR 1070: Could not resolve CassandraStorage2 using imports: [, > org.apache.pig.builtin., org.apache.pig.impl.builtin.] > > + PIG_OPTS=' -Dudf.import.list=org.apache.cassandra.hadoop.pig > -Dpig.additional.jars=/usr/local/Cellar/cassandra/1.2.4/apache-cassandra-thrift-1.2.4.jar:/usr/local/Cellar/cassandra/1.2.4/libthrift-0.7.0.jar > -Dpig.log.dir=/usr/local/bin/../Cellar/pig/0.11.1/bin/../logs' > + PIG_OPTS=' -Dudf.import.list=org.apache.cassandra.hadoop.pig > -Dpig.additional.jars=/usr/local/Cellar/cassandra/1.2.4/apache-cassandra-thrift-1.2.4.jar:/usr/local/Cellar/cassandra/1.2.4/libthrift-0.7.0.jar > -Dpig.log.dir=/usr/local/bin/../Cellar/pig/0.11.1/bin/../logs > -Dpig.log.file=pig.log' > + PIG_OPTS=' -Dudf.import.list=org.apache.cassandra.hadoop.pig > -Dpig.additional.jars=/usr/local/Cellar/cassandra/1.2.4/apache-cassandra-thrift-1.2.4.jar:/usr/local/Cellar/cassandra/1.2.4/libthrift-0.7.0.jar > -Dpig.log.dir=/usr/local/bin/../Cellar/pig/0.11.1/bin/../logs > -Dpig.log.file=pig.log > -Dpig.home.dir=/usr/local/bin/../Cellar/pig/0.11.1/bin/..' > + '[' '' == true ']' > + '[' -n /usr/local/bin/hadoop ']' > + '[' false == true ']' > + '[' -f /usr/local/bin/../Cellar/pig/0.11.1/bin/../pig-withouthadoop.jar > ']' > ++ echo > /usr/local/bin/../Cellar/pig/0.11.1/bin/../pig-0.11.1-withouthadoop.jar > + > PIG_JAR=/usr/local/bin/../Cellar/pig/0.11.1/bin/../pig-0.11.1-withouthadoop.jar > + '[' -z > /usr/local/bin/../Cellar/pig/0.11.1/bin/../pig-0.11.1-withouthadoop.jar ']' > + '[' -n > /usr/local/bin/../Cellar/pig/0.11.1/bin/../pig-0.11.1-withouthadoop.jar ']' > + > CLASSPATH='/usr/local/Cellar/hadoop/1.1.2/libexec/conf:/Library/Java/JavaVirtualMachines/1.6.0_32-b05-420.jdk/Contents/Home/lib/tools.jar:::/usr/local/Cellar/cassandra/1.2.4/*:/usr/local/Cellar/pig/0.11.1/pig-0.11.1.jar:/usr/local/bin/../Cellar/pig/0.11.1/bin/../pig-0.11.1-withouthadoop.jar' > + export > 'HADOOP_CLASSPATH=/usr/local/Cellar/hadoop/1.1.2/libexec/conf:/Library/Java/JavaVirtualMachines/1.6.0_32-b05-420.jdk/Contents/Home/lib/tools.jar:::/usr/local/Cellar/cassandra/1.2.4/*:/usr/local/Cellar/pig/0.11.1/pig-0.11.1.jar:/usr/local/bin/../Cellar/pig/0.11.1/bin/../pig-0.11.1-withouthadoop.jar:' > + > HADOOP_CLASSPATH='/usr/local/Cellar/hadoop/1.1.2/libexec/conf:/Library/Java/JavaVirtualMachines/1.6.0_32-b05-420.jdk/Contents/Home/lib/tools.jar:::/usr/local/Cellar/cassandra/1.2.4/*:/usr/local/Cellar/pig/0.11.1/pig-0.11.1.jar:/usr/local/bin/../Cellar/pig/0.11.1/bin/../pig-0.11.1-withouthadoop.jar:' > + export 'HADOOP_OPTS=-Xmx1000m > -Dudf.import.list=org.apache.cassandra.hadoop.pig > -Dpig.additional.jars=/usr/local/Cellar/cassandra/1.2.4/apache-cassandra-thrift-1.2.4.jar:/usr/local/Cellar/cassandra/1.2.4/libthrift-0.7.0.jar > -Dpig.log.dir=/usr/local/bin/../Cellar/pig/0.11.1/bin/../logs > -Dpig.log.file=pig.log > -Dpig.home.dir=/usr/local/bin/../Cellar/pig/0.11.1/bin/.. ' > + HADOOP_OPTS='-Xmx1000m > -Dudf.import.list=org.apache.cassandra.hadoop.pig > -Dpig.additional.jars=/usr/local/Cellar/cassandra/1.2.4/apache-cassandra-thrift-1.2.4.jar:/usr/local/Cellar/cassandra/1.2.4/libthrift-0.7.0.jar > -Dpig.log.dir=/usr/local/bin/../Cellar/pig/0.11.1/bin/../logs > -Dpig.log.file=pig.log > -Dpig.home.dir=/usr/local/bin/../Cellar/pig/0.11.1/bin/.. ' > + '[' false == true ']' > + exec /usr/local/bin/hadoop jar > /usr/local/bin/../Cellar/pig/0.11.1/bin/../pig-0.11.1-withouthadoop.jar > ./hadoop-sum-value_1.pig > 2013-05-03 09:57:56,810 [main] INFO org.apache.pig.Main - Apache Pig > version 0.11.1 (r1459641) compiled Mar 22 2013, 02:13:53 > 2013-05-03 09:57:56,811 [main] INFO org.apache.pig.Main - Logging error > messages to: /private/tmp/pig_1367567876808.log > 2013-05-03 09:57:56.980 java[81053:1203] Unable to load realm info from > SCDynamicStore > 2013-05-03 09:57:57,190 [main] INFO org.apache.pig.impl.util.Utils - > Default bootup file /Users/cyril/.pigbootup not found > 2013-05-03 09:57:57,333 [main] INFO > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - > Connecting to hadoop file system at: hdfs://localhost:9000 > 2013-05-03 09:57:57,513 [main] INFO > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - > Connecting to map-reduce job tracker at: localhost:9001 > 2013-05-03 09:57:57,944 [main] ERROR org.apache.pig.tools.grunt.Grunt - > ERROR 1070: Could not resolve CassandraStorage2 using imports: > [org.apache.cassandra.hadoop.pig., , org.apache.pig.builtin., > org.apache.pig.impl.builtin.] > > > I'm using an undefined function just to get the error which indicates the > import list which is not the same as you can see even if I use the same > options : > PIG_OPTS = ' -Dudf.import.list=org.apache.cassandra.hadoop.pig > -Dpig.additional.jars=/usr/local/Cellar/cassandra/1.2.4/apache-cassandra-thrift-1.2.4.jar:/usr/local/Cellar/cassandra/1.2.4/libthrift-0.7.0.jar > -Dpig.log.dir=/usr/local/bin/../Cellar/pig/0.11.1/bin/../logs' > > If anyone has an explanation ? > > thanks > -- > Cyril SCETBON > > > > _________________________________________________________________________________________________________________________ > > Ce message et ses pieces jointes peuvent contenir des informations > confidentielles ou privilegiees et ne doivent donc > pas etre diffuses, exploites ou copies sans autorisation. Si vous avez > recu ce message par erreur, veuillez le signaler > a l'expediteur et le detruire ainsi que les pieces jointes. Les messages > electroniques etant susceptibles d'alteration, > France Telecom - Orange decline toute responsabilite si ce message a ete > altere, deforme ou falsifie. Merci. > > This message and its attachments may contain confidential or privileged > information that may be protected by law; > they should not be distributed, used or copied without authorisation. > If you have received this email in error, please notify the sender and > delete this message and its attachments. > As emails may be altered, France Telecom - Orange is not liable for > messages that have been modified, changed or falsified. > Thank you. > > > > > _________________________________________________________________________________________________________________________ > > Ce message et ses pieces jointes peuvent contenir des informations > confidentielles ou privilegiees et ne doivent donc > pas etre diffuses, exploites ou copies sans autorisation. Si vous avez > recu ce message par erreur, veuillez le signaler > a l'expediteur et le detruire ainsi que les pieces jointes. Les messages > electroniques etant susceptibles d'alteration, > France Telecom - Orange decline toute responsabilite si ce message a ete > altere, deforme ou falsifie. Merci. > > This message and its attachments may contain confidential or privileged > information that may be protected by law; > they should not be distributed, used or copied without authorisation. > If you have received this email in error, please notify the sender and > delete this message and its attachments. > As emails may be altered, France Telecom - Orange is not liable for > messages that have been modified, changed or falsified. > Thank you. > >
