Ok, it just seems to be an issue with the syntax of the spark-submit command.
It should be :
spark-submit --queue default \
--class com.my.Launcher \
--deploy-mode cluster \
--master yarn-cluster \
--driver-java-options "-Dfile.encoding=UTF-8" \
--jars /home/user/hibernate-validator-5.2.2.Final.j
I've included that in my build file for the fat jar already.
libraryDependencies += "com.amazonaws" % "aws-java-sdk" % "1.11.155"
libraryDependencies += "com.amazonaws" % "aws-java-sdk-s3" % "1.11.155"
libraryDependencies += "com.amazonaws" % "aws-java-sdk-core" % "1.11.155"
Not sure if I need s
Ensure com.amazonaws.services.s3.AmazonS3ClientBuilder in your classpath
which include your application jar and attached executor jars.
2017-07-20 6:12 GMT+08:00 Noppanit Charassinvichai :
> I have this spark job which is using S3 client in mapPartition. And I get
> this error
>
> Job aborted due
Thanks Marcelo.
Problem solved.
Best,
Carlo
Hi Marcelo,
Thanks you for your help.
Problem solved as you suggested.
Best Regards,
Carlo
> On 5 Aug 2016, at 18:34, Marcelo Vanzin wrote:
>
> On Fri, Aug 5, 2016 at 9:53 AM, Carlo.Allocca
> wrote:
>>
>>org.apache.spark
>>
I have also executed:
mvn dependency:tree |grep log
[INFO] | | +- com.esotericsoftware:minlog:jar:1.3.0:compile
[INFO] +- log4j:log4j:jar:1.2.17:compile
[INFO] +- org.slf4j:slf4j-log4j12:jar:1.7.16:compile
[INFO] | | +- commons-logging:commons-logging:jar:1.1.3:compile
and the POM reports
On Fri, Aug 5, 2016 at 9:53 AM, Carlo.Allocca wrote:
>
> org.apache.spark
> spark-core_2.10
> 2.0.0
> jar
>
>
> org.apache.spark
> spark-sql_2.10
> 2.0.0
> jar
>
>
Please Sean, could you detail the version mismatch?
Many thanks,
Carlo
On 5 Aug 2016, at 18:11, Sean Owen
mailto:so...@cloudera.com>> wrote:
You also seem to have a
version mismatch here.
-- The Open University is incorporated by Royal Charter (RC 000391), an exempt
charity in England & Wales
Logging has always been an internal Spark class, and is marked as such
in later versions. You shouldn't use it. You also seem to have a
version mismatch here.
On Fri, Aug 5, 2016 at 10:10 AM, Carlo.Allocca wrote:
> Hi Ted,
>
> Thanks for the promptly answer.
> It is not yet clear to me what I sho
One option is to clone the class in your own project.
Experts may have better solution.
Cheers
On Fri, Aug 5, 2016 at 10:10 AM, Carlo.Allocca
wrote:
> Hi Ted,
>
> Thanks for the promptly answer.
> It is not yet clear to me what I should do.
>
> How to fix it?
>
> Many thanks,
> Carlo
>
> On 5
Hi Ted,
Thanks for the promptly answer.
It is not yet clear to me what I should do.
How to fix it?
Many thanks,
Carlo
On 5 Aug 2016, at 17:58, Ted Yu
mailto:yuzhih...@gmail.com>> wrote:
private[spark] trait Logging {
-- The Open University is incorporated by Royal Charter (RC 000391), an exe
In 2.0, Logging became private:
private[spark] trait Logging {
FYI
On Fri, Aug 5, 2016 at 9:53 AM, Carlo.Allocca
wrote:
> Dear All,
>
> I would like to ask for your help about the following issue:
> java.lang.ClassNotFoundException:
> org.apache.spark.Logging
>
> I checked and the class Loggi
Can you try running the example like this
./bin/run-example sql.RDDRelation
I know there are some jars in the example folders, and running them this
way adds them to the classpath
On Jul 7, 2016 3:47 AM, "kevin" wrote:
> hi,all:
> I build spark use:
>
> ./make-distribution.sh --name "hadoop2.7
Thanks Jacob,
I've looked into the source code here and found that I miss this property
there:
spark.repl.class.uri
Putting it solved the problem
Cheers
2016-03-17 18:14 GMT-03:00 Jakob Odersky :
> The error is very strange indeed, however without code that reproduces
> it, we can't really prov
The error is very strange indeed, however without code that reproduces
it, we can't really provide much help beyond speculation.
One thing that stood out to me immediately is that you say you have an
RDD of Any where every Any should be a BigDecimal, so why not specify
that type information?
When
Hi Ted, thanks for answering.
The map is just that, whenever I try inside the map it throws this
ClassNotFoundException, even if I do map(f => f) it throws the exception.
What is bothering me is that when I do a take or a first it returns the
result, which make me conclude that the previous code is
bq. $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1
Do you mind showing more of your code involving the map() ?
On Thu, Mar 17, 2016 at 8:32 AM, Dirceu Semighini Filho <
dirceu.semigh...@gmail.com> wrote:
> Hello,
> I found a strange behavior after executing a prediction with MLIB.
> My code
I found out that by commenting this line in the application code:
sparkConf.set("spark.executor.extraJavaOptions", " -XX:+UseCompressedOops
-XX:+UseConcMarkSweepGC -XX:+AggressiveOpts -XX:FreqInlineSize=300
-XX:MaxInlineSize=300 ")
the exception does not occur anymore. Not entirely sure why, but
you need make sure this class is accessible to all servers since its a
cluster mode and drive can be on any of the worker nodes.
On Fri, Dec 25, 2015 at 5:57 PM, Saiph Kappa wrote:
> Hi,
>
> I'm submitting a spark job like this:
>
> ~/spark-1.5.2-bin-hadoop2.6/bin/spark-submit --class Benchmark
I'm not %100 sure, but I don't think a jar within a jar will work without a
custom class loader. You can perhaps try to use "maven-assembly-plugin" or
"maven-shade-plugin" to build your uber/fat jar. Both of these will build a
flattened single jar.
--
Ali
On Nov 26, 2015, at 2:49 AM, Marc de
It turned out to be a problem with `SerializationUtils` from Apache Commons
Lang. There is an open issue where the class will throw a
`ClassNotFoundException` even if the class is in the classpath in a
multiple-classloader environment:
https://issues.apache.org/jira/browse/LANG-1049
We moved away
Where is the exception thrown (full stack trace)? How are you running your
application, via spark-submit or spark-shell?
On Tue, Nov 3, 2015 at 1:43 AM, hveiga wrote:
> Hello,
>
> I am facing an issue where I cannot run my Spark job in a cluster
> environment (standalone or EMR) but it works suc
Now I am running up against some other problem while trying to schedule tasks:
15/05/01 22:32:03 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
java.lang.IllegalStateException: unread block data
at
java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.j
I cherry-picked the fix for SPARK-5470 and the problem has gone away.
On Fri, May 1, 2015 at 9:15 AM, Akshat Aranya wrote:
> Yes, this class is present in the jar that was loaded in the classpath
> of the executor Java process -- it wasn't even lazily added as a part
> of the task execution. Sch
Yes, this class is present in the jar that was loaded in the classpath
of the executor Java process -- it wasn't even lazily added as a part
of the task execution. Schema$MyRow is a protobuf-generated class.
After doing some digging around, I think I might be hitting up against
SPARK-5470, the fi
bq. Caused by: java.lang.ClassNotFoundException: com.example.Schema$MyRow
So the above class is in the jar which was in the classpath ?
Can you tell us a bit more about Schema$MyRow ?
On Fri, May 1, 2015 at 8:05 AM, Akshat Aranya wrote:
> Hi,
>
> I'm getting a ClassNotFoundException at the exec
Hi Kevin,
yes I can test it means I have to build Spark from git repository?
Ralph
Am 17.03.15 um 02:59 schrieb Kevin (Sangwoo) Kim:
> Hi Ralph,
>
> It seems like https://issues.apache.org/jira/browse/SPARK-6299 issue,
> which is I'm working on.
> I submitted a PR for it, would you test
Hi Ralph,
It seems like https://issues.apache.org/jira/browse/SPARK-6299 issue, which
is I'm working on.
I submitted a PR for it, would you test it?
Regards,
Kevin
On Tue, Mar 17, 2015 at 1:11 AM Ralph Bergmann wrote:
> Hi,
>
>
> I want to try the JavaSparkPi example[1] on a remote Spark serve
Thanks for the notification!
For now, I'll use the Kryo serializer without registering classes until the
bug fix has been merged into the next version of Spark (I guess that will
be 1.3, right?).
arun
On Sun, Feb 1, 2015 at 10:58 PM, Shixiong Zhu wrote:
> It's a bug that has been fixed in
> h
It's a bug that has been fixed in https://github.com/apache/spark/pull/4258
but not yet been merged.
Best Regards,
Shixiong Zhu
2015-02-02 10:08 GMT+08:00 Arun Lists :
> Here is the relevant snippet of code in my main program:
>
> ===
>
> sparkConf.set("sp
asicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
View this message in context: Re
the Worker where the
>> ClassNotFoundException happens but it is a bit painful
>> This message and the information contained herein is proprietary and
>> confidential and subject to the Amdocs policy statement, you may review at
>> http://www.amdocs.com/email_disclaime
Can you make sure the class "SimpleApp$$anonfun$1" is included in your app
jar?
2014-11-20 18:19 GMT+01:00 Benoit Pasquereau [via Apache Spark User List] <
ml-node+s1001560n19391...@n3.nabble.com>:
> Hi Guys,
>
>
>
> I’m having an issue in standalone mode (Spark 1.1, Hadoop 2.4, Windows
> Server
As suggested, I opened this Jira:
https://issues.apache.org/jira/browse/SPARK-2576
S
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/ClassNotFoundException-line11-read-when-loading-an-HDFS-text-file-with-SparkQL-in-spark-shell-tp9954p10155.html
Sent from
Hi,
Yes, the error still occurs when we replace the lambdas with named
functions:
(same error traces as in previous posts)
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/ClassNotFoundException-line11-read-when-loading-an-HDFS-text-file-with-SparkQL-i
H, it could be some weirdness with classloaders / Mesos / spark sql?
I'm curious if you would hit an error if there were no lambda functions
involved. Perhaps if you load the data using jsonFile or parquetFile.
Either way, I'd file a JIRA. Thanks!
On Jul 16, 2014 6:48 PM, "Svend" wrote:
>
Hi Michael,
Thanks for your reply. Yes, the reduce triggered the actual execution, I got
a total length (totalLength: 95068762, for the record).
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/ClassNotFoundException-line11-read-when-loading-an-HDFS-text
Oh, I'm sorry... reduce is also an operation
On Wed, Jul 16, 2014 at 3:37 PM, Michael Armbrust
wrote:
>
> Note that runnning a simple map+reduce job on the same hdfs files with the
>> same installation works fine:
>>
>
> Did you call collect() on the totalLength? Otherwise nothing has
> ac
> Note that runnning a simple map+reduce job on the same hdfs files with the
> same installation works fine:
>
Did you call collect() on the totalLength? Otherwise nothing has actually
executed.
Hi Andrew,
Thanks for the current doc.
I'd almost gotten to the point where I thought that my custom code needed
> to be included in the SPARK_EXECUTOR_URI but that can't possibly be
> correct. The Spark workers that are launched on Mesos slaves should start
> with the Spark core jars and then t
Here's the 1.0.0rc9 version of the docs:
https://people.apache.org/~pwendell/spark-1.0.0-rc9-docs/running-on-mesos.html
I refreshed them with the goal of steering users more towards prebuilt
packages than relying on compiling from source plus improving overall
formatting and clarity, but not otherw
Hi Tobias,
On Wed, May 21, 2014 at 5:45 PM, Tobias Pfeiffer wrote:
>first, thanks for your explanations regarding the jar files!
No prob :-)
> On Thu, May 22, 2014 at 12:32 AM, Gerard Maas
> wrote:
> > I was discussing it with my fellow Sparkers here and I totally overlooked
> > the fact that
Hi Gerard,
first, thanks for your explanations regarding the jar files!
On Thu, May 22, 2014 at 12:32 AM, Gerard Maas wrote:
> I was discussing it with my fellow Sparkers here and I totally overlooked
> the fact that you need the class files to de-serialize the closures (or
> whatever) on the wo
Hi Tobias,
Regarding my comment on closure serialization:
I was discussing it with my fellow Sparkers here and I totally overlooked
the fact that you need the class files to de-serialize the closures (or
whatever) on the workers, so you always need the jar file delivered to the
workers in order f
Hi Tobias,
For your simple example, I just used sbt package, but for more complex jobs
that have external dependencies, either:
- you should use sbt assembly [1] or mvn shade plugin [2] to build a "fat
jar" (aka jar-with-dependencies)
- or provide a list of jars including your job jar along wit
Gerard,
thanks very much for your investigation! After hours of trial and
error, I am kind of happy to hear it is not just a broken setup on my
side that's causing the error.
Could you explain briefly how you created that simple jar file?
Thanks,
Tobias
On Wed, May 21, 2014 at 9:47 PM, Gerard M
Hi Tobias,
I was curious about this issue and tried to run your example on my local
Mesos. I was able to reproduce your issue using your current config:
[error] (run-main-0) org.apache.spark.SparkException: Job aborted: Task
1.0:4 failed 4 times (most recent failure: Exception failure:
java.lang.
I just ran into the same problem. I will respond if I find how to fix.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/ClassNotFoundException-tp5182p5342.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Please help me
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/ClassNotFoundException-tp5182p5209.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
48 matches
Mail list logo
