Hi Cheng, I replaced Guava 15.0 with Guava 14.0.1 in my spark classpath, the problem was solved. So your method is correct. It proved that this issue was caused by AWS EMR (ami-version 3.1.0) libs which include Guava 15.0.
Many thanks and see you in the first Spark User Beijing Meetup tomorrow. -- Zhun Shen Data Mining at LightnInTheBox.com Email: shenzhunal...@gmail.com | shenz...@yahoo.com Phone: 186 0627 7769 GitHub: https://github.com/shenzhun LinkedIn: http://www.linkedin.com/in/shenzhun On August 7, 2014 at 6:57:06 PM, Cheng Lian (lian.cs....@gmail.com) wrote: Hey Zhun, Thanks for the detailed problem description. Please see my comments inlined below. On Thu, Aug 7, 2014 at 6:18 PM, Zhun Shen <shenzhunal...@gmail.com> wrote: Caused by: java.lang.IllegalAccessError: tried to access method com.google.common.collect.MapMaker.makeComputingMap(Lcom/google/common/base/Function;)Ljava/util/concurrent/ConcurrentMap; from class com.jolbox.bonecp.BoneCPDataSource This line indicates that accessing MapMaker.makeComputingMap via Java reflection fails. The version of Guava we used in Spark SQL (as a transitive dependency) is 14.0.1. In this version, MapMaker.makeComputingMap is still public. But in newer versions (say 15.0), it’s no longer public. So my guess is that, a newer version of the Guava library in your classpath shadows the version Spark SQL uses somehow. A quick and dirty fix to see whether this is true is try putting Guava 14.0.1 jar file at the beginning of your classpath and see whether things work. at com.jolbox.bonecp.BoneCPDataSource.<init>(BoneCPDataSource.java:64) at org.datanucleus.store.rdbms.datasource.BoneCPDataSourceFactory.makePooledDataSource(BoneCPDataSourceFactory.java:73) at org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:217) at org.datanucleus.store.rdbms.ConnectionFactoryImpl.initialiseDataSources(ConnectionFactoryImpl.java:110) at org.datanucleus.store.rdbms.ConnectionFactoryImpl.<init>(ConnectionFactoryImpl.java:82) ... 119 more
