I think it's a bug in the default hadoop-env.sh : # Extra Java runtime options. Empty by default. export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true $HADOOP_CLIENT_OPTS"
we should be able to overload this in the console without having to go into hadoop-env.sh to change it. So I replaced this line by : # Extra Java runtime options. Empty by default. export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true $HADOOP_CLIENT_OPTS $HADOOP_OPTS" Regards -- Cyril SCETBON On May 6, 2013, at 7:35 PM, Prashant Kommireddi <[email protected]<mailto:[email protected]>> wrote: This blog post describes some of this behavior http://hadoopified.wordpress.com/2013/04/07/pig-startup-script-behavior/ On Mon, May 6, 2013 at 1:07 AM, <[email protected]> wrote: forget it ! It came from hadoop-env.sh provided by the package I installed on one cluster :( Regards -- Cyril SCETBON On May 3, 2013, at 11:44 AM, [email protected]<mailto: [email protected]> wrote: Hi, I have 2 differents behaviour for the same Pig version (with hadoop 1.1.2) on different servers. If anyone can tell me why with the same versions and the same parameters I don't have the same behaviour. I'm trying to set the builtin import list but it's not used with one hadoop cluster whereas it's used when I use it on a single node cluster : + PIG_OPTS=' -Dudf.import.list=org.apache.cassandra.hadoop.pig -Dpig.additional.jars=/usr/local/Cellar/cassandra/1.2.4/apache-cassandra-thrift-1.2.4.jar:/usr/local/Cellar/cassandra/1.2.4/libthrift-0.7.0.jar -Dpig.log.dir=/usr/local/bin/../Cellar/pig/0.11.1/bin/../logs' + PIG_OPTS=' -Dudf.import.list=org.apache.cassandra.hadoop.pig -Dpig.additional.jars=/usr/local/Cellar/cassandra/1.2.4/apache-cassandra-thrift-1.2.4.jar:/usr/local/Cellar/cassandra/1.2.4/libthrift-0.7.0.jar -Dpig.log.dir=/usr/local/bin/../Cellar/pig/0.11.1/bin/../logs -Dpig.log.file=pig.log' + PIG_OPTS=' -Dudf.import.list=org.apache.cassandra.hadoop.pig -Dpig.additional.jars=/usr/local/Cellar/cassandra/1.2.4/apache-cassandra-thrift-1.2.4.jar:/usr/local/Cellar/cassandra/1.2.4/libthrift-0.7.0.jar -Dpig.log.dir=/usr/local/bin/../Cellar/pig/0.11.1/bin/../logs -Dpig.log.file=pig.log -Dpig.home.dir=/usr/local/bin/../Cellar/pig/0.11.1/bin/..' + '[' '' == true ']' + '[' -n /usr/local/bin/hadoop ']' + '[' false == true ']' + '[' -f /usr/local/bin/../Cellar/pig/0.11.1/bin/../pig-withouthadoop.jar ']' ++ echo /usr/local/bin/../Cellar/pig/0.11.1/bin/../pig-0.11.1-withouthadoop.jar + PIG_JAR=/usr/local/bin/../Cellar/pig/0.11.1/bin/../pig-0.11.1-withouthadoop.jar + '[' -z /usr/local/bin/../Cellar/pig/0.11.1/bin/../pig-0.11.1-withouthadoop.jar ']' + '[' -n /usr/local/bin/../Cellar/pig/0.11.1/bin/../pig-0.11.1-withouthadoop.jar ']' + CLASSPATH='/etc/hadoop/conf:/usr/lib/jvm/java-7-oracle/lib/tools.jar:::/opt/apache-cassandra-1.2.3/lib/*:/usr/share/hadoop/lib/*:/opt/pig-0.11.1/pig-0.11.1-withouthadoop.jar:/etc/hadoop:/opt/pig-0.11.1/bin/../lib/jython-standalone-2.5.3.jar:/opt/pig-0.11.1/bin/../pig-0.11.1-withouthadoop.jar' + export 'HADOOP_CLASSPATH=/etc/hadoop/conf:/usr/lib/jvm/java-7-oracle/lib/tools.jar:::/opt/apache-cassandra-1.2.3/lib/*:/usr/share/hadoop/lib/*:/opt/pig-0.11.1/pig-0.11.1-withouthadoop.jar:/etc/hadoop:/opt/pig-0.11.1/bin/../lib/jython-standalone-2.5.3.jar:/opt/pig-0.11.1/bin/../pig-0.11.1-withouthadoop.jar:' + HADOOP_CLASSPATH='/etc/hadoop/conf:/usr/lib/jvm/java-7-oracle/lib/tools.jar:::/opt/apache-cassandra-1.2.3/lib/*:/usr/share/hadoop/lib/*:/opt/pig-0.11.1/pig-0.11.1-withouthadoop.jar:/etc/hadoop:/opt/pig-0.11.1/bin/../lib/jython-standalone-2.5.3.jar:/opt/pig-0.11.1/bin/../pig-0.11.1-withouthadoop.jar:' + export 'HADOOP_OPTS=-Xmx1000m -Dudf.import.list=org.apache.cassandra.hadoop.pig -Dpig.additional.jars=/opt/apache-cassandra-1.2.3/lib/apache-cassandra-thrift-1.2.3.jar:/opt/apache-cassandra-1.2.3/lib/libthrift-0.7.0.jar -4 log4j.properties -Dpig.log.dir=/opt/pig-0.11.1/bin/../logs -Dpig.log.file=pig.log -Dpig.home.dir=/opt/pig-0.11.1/bin/.. -Djava.net.preferIPv4Stack=true ' + HADOOP_OPTS='-Xmx1000m -Dudf.import.list=org.apache.cassandra.hadoop.pig -Dpig.additional.jars=/opt/apache-cassandra-1.2.3/lib/apache-cassandra-thrift-1.2.3.jar:/opt/apache-cassandra-1.2.3/lib/libthrift-0.7.0.jar -4 log4j.properties -Dpig.log.dir=/opt/pig-0.11.1/bin/../logs -Dpig.log.file=pig.log -Dpig.home.dir=/opt/pig-0.11.1/bin/.. -Djava.net.preferIPv4Stack=true ' + '[' false == true ']' + exec /usr/bin/hadoop jar /opt/pig-0.11.1/bin/../pig-0.11.1-withouthadoop.jar ./hadoop-sum-value_1.pig 2013-05-03 07:57:55,140 [main] INFO org.apache.pig.Main - Apache Pig version 0.11.1 (r1459641) compiled Mar 22 2013, 02:13:53 2013-05-03 07:57:55,141 [main] INFO org.apache.pig.Main - Logging error messages to: /home/ubuntu/pig_1367567875107.log 2013-05-03 07:57:57,502 [main] INFO org.apache.pig.impl.util.Utils - Default bootup file /home/ubuntu/.pigbootup not found 2013-05-03 07:57:58,461 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://hadoop-cass-1:9100 2013-05-03 07:57:59,663 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to map-reduce job tracker at: hadoop-cass-1:8021 2013-05-03 07:58:01,387 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1070: Could not resolve CassandraStorage2 using imports: [, org.apache.pig.builtin., org.apache.pig.impl.builtin.] + PIG_OPTS=' -Dudf.import.list=org.apache.cassandra.hadoop.pig -Dpig.additional.jars=/usr/local/Cellar/cassandra/1.2.4/apache-cassandra-thrift-1.2.4.jar:/usr/local/Cellar/cassandra/1.2.4/libthrift-0.7.0.jar -Dpig.log.dir=/usr/local/bin/../Cellar/pig/0.11.1/bin/../logs' + PIG_OPTS=' -Dudf.import.list=org.apache.cassandra.hadoop.pig -Dpig.additional.jars=/usr/local/Cellar/cassandra/1.2.4/apache-cassandra-thrift-1.2.4.jar:/usr/local/Cellar/cassandra/1.2.4/libthrift-0.7.0.jar -Dpig.log.dir=/usr/local/bin/../Cellar/pig/0.11.1/bin/../logs -Dpig.log.file=pig.log' + PIG_OPTS=' -Dudf.import.list=org.apache.cassandra.hadoop.pig -Dpig.additional.jars=/usr/local/Cellar/cassandra/1.2.4/apache-cassandra-thrift-1.2.4.jar:/usr/local/Cellar/cassandra/1.2.4/libthrift-0.7.0.jar -Dpig.log.dir=/usr/local/bin/../Cellar/pig/0.11.1/bin/../logs -Dpig.log.file=pig.log -Dpig.home.dir=/usr/local/bin/../Cellar/pig/0.11.1/bin/..' + '[' '' == true ']' + '[' -n /usr/local/bin/hadoop ']' + '[' false == true ']' + '[' -f /usr/local/bin/../Cellar/pig/0.11.1/bin/../pig-withouthadoop.jar ']' ++ echo /usr/local/bin/../Cellar/pig/0.11.1/bin/../pig-0.11.1-withouthadoop.jar + PIG_JAR=/usr/local/bin/../Cellar/pig/0.11.1/bin/../pig-0.11.1-withouthadoop.jar + '[' -z /usr/local/bin/../Cellar/pig/0.11.1/bin/../pig-0.11.1-withouthadoop.jar ']' + '[' -n /usr/local/bin/../Cellar/pig/0.11.1/bin/../pig-0.11.1-withouthadoop.jar ']' + CLASSPATH='/usr/local/Cellar/hadoop/1.1.2/libexec/conf:/Library/Java/JavaVirtualMachines/1.6.0_32-b05-420.jdk/Contents/Home/lib/tools.jar:::/usr/local/Cellar/cassandra/1.2.4/*:/usr/local/Cellar/pig/0.11.1/pig-0.11.1.jar:/usr/local/bin/../Cellar/pig/0.11.1/bin/../pig-0.11.1-withouthadoop.jar' + export 'HADOOP_CLASSPATH=/usr/local/Cellar/hadoop/1.1.2/libexec/conf:/Library/Java/JavaVirtualMachines/1.6.0_32-b05-420.jdk/Contents/Home/lib/tools.jar:::/usr/local/Cellar/cassandra/1.2.4/*:/usr/local/Cellar/pig/0.11.1/pig-0.11.1.jar:/usr/local/bin/../Cellar/pig/0.11.1/bin/../pig-0.11.1-withouthadoop.jar:' + HADOOP_CLASSPATH='/usr/local/Cellar/hadoop/1.1.2/libexec/conf:/Library/Java/JavaVirtualMachines/1.6.0_32-b05-420.jdk/Contents/Home/lib/tools.jar:::/usr/local/Cellar/cassandra/1.2.4/*:/usr/local/Cellar/pig/0.11.1/pig-0.11.1.jar:/usr/local/bin/../Cellar/pig/0.11.1/bin/../pig-0.11.1-withouthadoop.jar:' + export 'HADOOP_OPTS=-Xmx1000m -Dudf.import.list=org.apache.cassandra.hadoop.pig -Dpig.additional.jars=/usr/local/Cellar/cassandra/1.2.4/apache-cassandra-thrift-1.2.4.jar:/usr/local/Cellar/cassandra/1.2.4/libthrift-0.7.0.jar -Dpig.log.dir=/usr/local/bin/../Cellar/pig/0.11.1/bin/../logs -Dpig.log.file=pig.log -Dpig.home.dir=/usr/local/bin/../Cellar/pig/0.11.1/bin/.. ' + HADOOP_OPTS='-Xmx1000m -Dudf.import.list=org.apache.cassandra.hadoop.pig -Dpig.additional.jars=/usr/local/Cellar/cassandra/1.2.4/apache-cassandra-thrift-1.2.4.jar:/usr/local/Cellar/cassandra/1.2.4/libthrift-0.7.0.jar -Dpig.log.dir=/usr/local/bin/../Cellar/pig/0.11.1/bin/../logs -Dpig.log.file=pig.log -Dpig.home.dir=/usr/local/bin/../Cellar/pig/0.11.1/bin/.. ' + '[' false == true ']' + exec /usr/local/bin/hadoop jar /usr/local/bin/../Cellar/pig/0.11.1/bin/../pig-0.11.1-withouthadoop.jar ./hadoop-sum-value_1.pig 2013-05-03 09:57:56,810 [main] INFO org.apache.pig.Main - Apache Pig version 0.11.1 (r1459641) compiled Mar 22 2013, 02:13:53 2013-05-03 09:57:56,811 [main] INFO org.apache.pig.Main - Logging error messages to: /private/tmp/pig_1367567876808.log 2013-05-03 09:57:56.980 java[81053:1203] Unable to load realm info from SCDynamicStore 2013-05-03 09:57:57,190 [main] INFO org.apache.pig.impl.util.Utils - Default bootup file /Users/cyril/.pigbootup not found 2013-05-03 09:57:57,333 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://localhost:9000 2013-05-03 09:57:57,513 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to map-reduce job tracker at: localhost:9001 2013-05-03 09:57:57,944 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1070: Could not resolve CassandraStorage2 using imports: [org.apache.cassandra.hadoop.pig., , org.apache.pig.builtin., org.apache.pig.impl.builtin.] I'm using an undefined function just to get the error which indicates the import list which is not the same as you can see even if I use the same options : PIG_OPTS = ' -Dudf.import.list=org.apache.cassandra.hadoop.pig -Dpig.additional.jars=/usr/local/Cellar/cassandra/1.2.4/apache-cassandra-thrift-1.2.4.jar:/usr/local/Cellar/cassandra/1.2.4/libthrift-0.7.0.jar -Dpig.log.dir=/usr/local/bin/../Cellar/pig/0.11.1/bin/../logs' If anyone has an explanation ? thanks -- Cyril SCETBON _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, France Telecom - Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, France Telecom - Orange is not liable for messages that have been modified, changed or falsified. Thank you. _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, France Telecom - Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, France Telecom - Orange is not liable for messages that have been modified, changed or falsified. Thank you. _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, France Telecom - Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, France Telecom - Orange is not liable for messages that have been modified, changed or falsified. Thank you.
