ClassNotFoundException while unmarshalling a remote RDD on Spark 1.5.1

Hi ! I'm facing a Classloader problem using Spark 1.5.1 I use javax.validation and hibernate validation annotations on some of my beans : @NotBlank @Valid private String attribute1 ; @Valid private String attribute2 ; When Spark tries to unmarshall these beans (after a remote RDD),

Spark UI port

Hello all, I am running PySpark Job (v2.0.2) with checkpoint enabled in Mesos cluster and am using Marathon for orchestration. When the job is restarted using Marathon, Spark UI is not getting started at the port specified by Marathon. Instead, it is picking port from the checkpoint. Is there a

Re: How to convert Row to JSON in Java?

Sorry - more likely l700 save. jg > On Sep 10, 2017, at 20:56, Jean Georges Perrin wrote: > > Hey, > > I have a few examples https://github.com/jgperrin/net.jgp.labs.spark. I > recently worked on such problems, so there's definitely a solution there or > I'll be happy to write one for you.

Re: How to convert Row to JSON in Java?

Hey, I have a few examples https://github.com/jgperrin/net.jgp.labs.spark. I recently worked on such problems, so there's definitely a solution there or I'll be happy to write one for you. Look in l250 map... jg > On Sep 10, 2017, at 20:51, ayan guha wrote: > > Sorry for side-line questi

Re: How to convert Row to JSON in Java?

Sorry for side-line question, but for Python, isn't following the easiest: >>> import json >>>* df1 = df.rdd.map(lambda r: json.dumps(r.asDict()))* >>> df1.take(10) ['{"id": 1}', '{"id": 2}', '{"id": 3}', '{"id": 4}', '{"id": 5}'] On Mon, Sep 11, 2017 at 4:22 AM, Riccardo Ferrari wrote: > Hi

Re: Multiple vcores per container when running Spark applications in Yarn cluster mode

I guess you're using Capacity Scheduler with DefaultResourceCalculator, which doesn't count cpu cores into resource calculation, this "1" you saw is actually meaningless. If you want to also calculate cpu resource, you should choose DominantResourceCalculator. Thanks Jerry On Sat, Sep 9, 2017 at

Re: Bizarre UI Behavior after migration

Was checking mails I sent, and wanted to get back to this one in case someone gets the same question. We found out that the reason why we saw stages being complete without all tasks complete is related to issues in the ListenerBus We had to tune the event queue size, see this https://issues.apach

Re: How to convert Row to JSON in Java?

Hi Kant, You can check the getValuesMap . I found this post useful, it is in Scala but should be a good starting point. An alternative appro

[Spark Streaming] - Stopped worker throws FileNotFoundException

I am running a spark streaming application on a cluster composed by three nodes, each one with a worker and three executors (so a total of 9 executors). I am using the spark standalone mode (version 2.1.1). The application is run with a spark-submit command with option "--deploy-mode" client and "

Re: Spark standalone API...

Hello, you might get the information you are looking for from this hidden API: http://:/json/ Hope it helps, Davide -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ - To unsubscribe e-mail: user-uns