Its a jar conflict (http-client <http://mvnrepository.com/artifact/org.apache.httpcomponents/httpclient/4.0-alpha4> jar), You could download the appropriate version of that jar and put that in the classpath before your assembly jar and hopefully it will avoid the conflict.
Thanks Best Regards On Thu, Dec 11, 2014 at 10:13 PM, Albert Manyà <alber...@eml.cc> wrote: > > Hi, > > I've made a simple script in scala that after doing a spark sql query it > sends the result to AWS's cloudwatch. > > I've tested both parts individually (the spark sql one and the > cloudwatch one) and they worked fine. The trouble comes when I execute > the script through spark-submit that gives me the following exception: > > Exception in thread "main" java.lang.NoSuchMethodError: > > org.apache.http.params.HttpConnectionParams.setSoKeepalive(Lorg/apache/http/params/HttpParams;Z)V > at > > com.amazonaws.http.HttpClientFactory.createHttpClient(HttpClientFactory.java:95) > at > > com.amazonaws.http.AmazonHttpClient.<init>(AmazonHttpClient.java:193) > at > > com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:120) > at > > com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:104) > at > > com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.<init>(AmazonCloudWatchClient.java:171) > at > > com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.<init>(AmazonCloudWatchClient.java:152) > at > > com.scmspain.synapse.SynapseMonitor$.sendMetrics(SynapseMonitor.scala:50) > at > com.scmspain.synapse.SynapseMonitor$.main(SynapseMonitor.scala:45) > at > com.scmspain.synapse.SynapseMonitor.main(SynapseMonitor.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:483) > at > org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:328) > at > org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > > my class is com.scmspain.synapse.SynapseMonitor > > I've build my script with sbt assembly, having the following > dependencies: > > libraryDependencies ++= Seq( > "org.apache.spark" %% "spark-core" % "1.1.0" % "provided", > "org.apache.spark" %% "spark-sql" % "1.1.0" % "provided", > "com.amazonaws" % "aws-java-sdk-cloudwatch" % "1.9.10" > ) > > I've unzipped the generated jar assembly and searched for the > HttpConnectionParams.class and I've found it out under > org/apache/http/params and having the following signature for > setSoKeepalive: > > public static void setSoKeepalive(HttpParams params, boolean > enableKeepalive) > > At this point I'm stuck and didn't know where to keep looking... some > help would be greatly appreciated :) > > Thank you very much! > > -- > Albert Manyà > alber...@eml.cc > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
