Thanks Dave. I found that Pig 0.14 and Hadoop 2.6.0 still use Guava 11.x
which was causing issue. I replacing all of those locations with Guava 17
did not end the ordeal. Seems like Guava made some breaking changes (
https://issues.apache.org/jira/browse/HADOOP-11032) in v17. You need
version 16.0 to be precise to get things working.
The MR job was still failing with ClassDefNotFound because it could not
find com.codahale.metrics.Metric in classpath, so I had to download and
include that. The example did run finally.
However, I still *cannot* use CqlStorage(), I had to rely on
CqlNativeStorage().
Anyway, thanks again.
On Fri, Jan 23, 2015 at 6:28 AM, Dave Brosius <dbros...@mebigfatguy.com>
wrote:
> The method
>
> com.google.common.collect.Sets.newConcurrentHashSet()Ljava/util/Set;
>
> should be available in guava from 15.0 on. So guava-16.0 should be fine.
>
> It's possible guava is being picked up from somewhere else? have a global
> classpath variable?
>
> you might want to do
>
> URL u = YourClass.getResource("/com/google/common/collect/Sets.class");
> System.out.println(u);
>
> to see where you are loading guava from.
>
>
> On 01/22/2015 04:12 AM, Pinak Pani wrote:
>
> I am using Pig with Cassandra (Cassandra 2.1.2, Pig 0.14, Hadoop 2.6.0
> combo).
>
> When I use CqlStorage() I get
>
> org.apache.pig.backend.executionengine.ExecException: ERROR 2118:
> org.apache.cassandra.exceptions.ConfigurationException: Unable to find
> inputformat class 'org.apache.cassandra.hadoop.cql3.CqlPagingInputFormat/
>
> When I use CqlNativeStorage() I get
>
> java.lang.NoSuchMethodError:
> com.google.common.collect.Sets.newConcurrentHashSet()Ljava/util/Set;
>
> Pig classpath looks like this:
>
> ยป echo $PIG_CLASSPATH
>
>
> /home/naishe/apps/apache-cassandra-2.1.2/lib/airline-0.6.jar:/home/naishe/apps/apache-cassandra-2.1.2/lib/antlr-runtime-3.5.2.jar:/home/naishe/apps/apache-cassandra-2.1.2/lib/apache-cassandra-2.1.2.jar:/home/naishe/apps/apache-cassandra-2.1.2/lib/apache-cassandra-clientutil-2.1.2.jar:/home/naishe/apps/apache-cassandra-2.1.2/lib/apache-cassandra-thrift-2.1.2.jar:/home/naishe/apps/apache-cassandra-2.1.2/lib/commons-cli-1.1.jar:/home/naishe/apps/apache-cassandra-2.1.2/lib/commons-codec-1.2.jar:/home/naishe/apps/apache-cassandra-2.1.2/lib/commons-lang3-3.1.jar:/home/naishe/apps/apache-cassandra-2.1.2/lib/commons-math3-3.2.jar:/home/naishe/apps/apache-cassandra-2.1.2/lib/compress-lzf-0.8.4.jar:/home/naishe/apps/apache-cassandra-2.1.2/lib/concurrentlinkedhashmap-lru-1.4.jar:/home/naishe/apps/apache-cassandra-2.1.2/lib/disruptor-3.0.1.jar:/home/naishe/apps/apache-cassandra-2.1.2/lib/
> *guava-16.0.jar*:/home/naishe/apps/apache-cassandra-2.1 .
> 2/lib/high-scale-lib-1.0.6.jar:/home/naishe/apps/apache-cassandra-2.1.2/lib/jackson-core-asl-1.9.2.jar:/home/naishe/apps/apache-cassandra-2.1.2/lib/jackson-mapper-asl-1.9.2.jar:/home/naishe/apps/apache-cassandra-2.1.2/lib/jamm-0.2.8.jar:/home/naishe/apps/apache-cassandra-2.1.2/lib/javax.inject.jar:/home/naishe/apps/apache-cassandra-2.1.2/lib/jbcrypt-0.3m.jar:/home/naishe/apps/apache-cassandra-2.1.2/lib/jline-1.0.jar:/home/naishe/apps/apache-cassandra-2.1.2/lib/jna-4.0.0.jar:/home/naishe/apps/apache-cassandra-2.1.2/lib/json-simple-1.1.jar:/home/naishe/apps/apache-cassandra-2.1.2/lib/libthrift-0.9.1.jar:/home/naishe/apps/apache-cassandra-2.1.2/lib/logback-classic-1.1.2.jar:/home/naishe/apps/apache-cassandra-2.1.2/lib/logback-core-1.1.2.jar:/home/naishe/apps/apache-cassandra-2.1.2/lib/lz4-1.2.0.jar:/home/naishe/apps/apache-cassandra-2.1.2/lib/metrics-core-2.2.0.jar:/home/naishe/apps/apache-cassandra-2.1.2/lib/netty-all-4.0.23.Final.jar:/home/naishe/apps/apache-cassandra-2.1.2/lib/report
> e
> r-config-2.1.0.jar:/home/naishe/apps/apache-cassandra-2.1.2/lib/slf4j-api-1.7.2.jar:/home/naishe/apps/apache-cassandra-2.1.2/lib/snakeyaml-1.11.jar:/home/naishe/apps/apache-cassandra-2.1.2/lib/snappy-java-1.0.5.2.jar:/home/naishe/apps/apache-cassandra-2.1.2/lib/stream-2.5.2.jar:/home/naishe/apps/apache-cassandra-2.1.2/lib/stringtemplate-4.0.2.jar:/home/naishe/apps/apache-cassandra-2.1.2/lib/super-csv-2.1.0.jar:/home/naishe/apps/apache-cassandra-2.1.2/lib/thrift-server-0.3.7.jar::/home/naishe/.m2/repository/com/datastax/cassandra/cassandra-driver-core/2.1.2/cassandra-driver-core-2.1.2.jar:/home/naishe/.m2/repository/org/apache/cassandra/cassandra-all/2.1.2/cassandra-all-2.1.2.jar
>
> I have read somewhere that it is due to version conflict with Guava
> library. So, I tried using Guava 11.0.2, that did not help. (
> http://stackoverflow.com/questions/27089126/nosuchmethoderror-sets-newconcurrenthashset-while-running-jar-using-hadoop#comment42687234_27089126
> )
>
> Here is the Pig latin that I was trying to execute.
>
> grunt> alice = LOAD 'cql://hadoop_test/lines' USING CqlNativeStorage();
>
> 2015-01-22 09:28:54,133 [main] INFO
> org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is
> deprecated. Instead, use fs.defaultFS
> grunt> B = foreach alice generate flatten(TOKENIZE((chararray)$0)) as word;
> grunt> C = group B by word;
> grunt> D = foreach C generate COUNT(B) as word_count, group as word;
> grunt> dump D;
> 2015-01-22 09:29:06,808 [main] INFO
> org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
> script: GROUP_BY
> [ -- snip -- ]
> 2015-01-22 09:29:11,254 [LocalJobRunner Map Task Executor #0] INFO
> org.apache.hadoop.mapred.MapTask - Map output collector class =
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer
> 2015-01-22 09:29:11,588 [LocalJobRunner Map Task Executor #0] INFO
> org.apache.hadoop.mapred.MapTask - Starting flush of map output
> 2015-01-22 09:29:11,600 [Thread-22] INFO
> org.apache.hadoop.mapred.LocalJobRunner - map task executor complete.
> 2015-01-22 09:29:11,620 [Thread-22] WARN
> org.apache.hadoop.mapred.LocalJobRunner - job_local1857630817_0001
> java.lang.Exception: java.lang.NoSuchMethodError:
> com.google.common.collect.Sets.newConcurrentHashSet()Ljava/util/Set;
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
> Caused by: java.lang.NoSuchMethodError:
> com.google.common.collect.Sets.newConcurrentHashSet()Ljava/util/Set;
> at
> com.datastax.driver.core.Cluster$ConnectionReaper.<init>(Cluster.java:2065)
> at com.datastax.driver.core.Cluster$Manager.<init>(Cluster.java:1163)
>
>
> However when I use
> alice = LOAD 'cql://hadoop_test/lines' USING CqlStorage();
> in above script,
>
> I get
> org.apache.pig.backend.executionengine.ExecException: ERROR 2118:
> org.apache.cassandra.exceptions.ConfigurationException: Unable to find
> inputformat class 'org.apache.cassandra.hadoop.cql3.CqlPagingInputFormat'
>
>
> Thanks for reading this mail.
>
>
>
