Exception while running unit tests that makes use of local-cluster mode

Hi All, When i try to run unit tests that makes use of local-cluster mode (Ex: "Accessing HttpBroadcast variables in a local cluster" in BroadcastSuite.scala), its failing with the below exception. I'm using java version "1.8.0_05" and scala version 2.10. I tried to look into the jenkins build re

Re: Development testing code

Hi, Many tests in pyspark are implemented as doctests and the python unittesting framework is also used for additional tests. Cheers, Holden :) On Wed, Oct 22, 2014 at 4:13 PM, catchmonster wrote: > Hi, > If developing in python, what is preffered way to do unit testing? > Do I use pyunit fra

Development testing code

Hi, If developing in python, what is preffered way to do unit testing? Do I use pyunit framework or I need to go with scalaTest? -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Development-testing-code-tp8911.html Sent from the Apache Spark Developers

Re: Multitenancy in Spark - within/across spark context

On Wed, Oct 22, 2014 at 2:17 PM, Ashwin Shankar wrote: >> That's not something you might want to do usually. In general, a >> SparkContext maps to a user application > > My question was basically this. In this page in the official doc, under > "Scheduling within an application" section, it talks a

Fwd: Sharing spark context across multiple spark sql cli initializations

We want to run multiple instances of spark sql cli on our yarn cluster. Each instance of the cli is to be used by a different user. This looks non-optimal if each user brings up a different cli given how spark works on yarn by running executor processes (and hence consuming resources) on worker nod

Sharing spark context across multiple spark sql cli initializations

We want to run multiple instances of spark sql cli on our yarn cluster. Each instance of the cli is to be used by a different user. This would be non-optimal if each user brings up a different cli given how spark works on yarn by running executor processes (and hence consuming resources) on worker

Re: Multitenancy in Spark - within/across spark context

Thanks Marcelo, that was helpful ! I had some follow up questions : That's not something you might want to do usually. In general, a > SparkContext maps to a user application My question was basically this. In this page in the official doc

Re: Multitenancy in Spark - within/across spark context

Hi Ashwin, Let me try to answer to the best of my knowledge. On Wed, Oct 22, 2014 at 11:47 AM, Ashwin Shankar wrote: > Here are my questions : > 1. Sharing spark context : How exactly multiple users can share the cluster > using same spark > context ? That's not something you might want to

Multitenancy in Spark - within/across spark context

Hi Spark devs/users, One of the things we are investigating here at Netflix is if Spark would suit us for our ETL needs, and one of requirements is multi tenancy. I did read the official doc and the book, but I'm still not clear on certain t

Re: Which part of the code deals with communication?

The best documentation about communication interfaces is the SecurityManager doc written by Tom Graves. With this as a starting point I'd recommend digging through the code for each component. https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SecurityManager.scala#L5

SPARK-3299 jira task question

Since this task involves making changes to some of core functionality, I figured it's best if I share my intents for completing this task. This change is a little more involved as it requires modifying the Catalog trait. My current plan is to add an abstract method to the Catalog trait and have an

Re: Easy win: SBT plugin config expert to help on SPARK-3359?

Hi Sean, I've pushed a PR for this https://github.com/apache/spark/pull/2893 :) Cheers, Holden :) On Tue, Oct 21, 2014 at 4:41 AM, Sean Owen wrote: > This one can be resolved, I think, with a bit of help from someone who > understands SBT + plugin config: > > https://issues.apache.org/jira/br

Re: Graphx connectComponents API

Well - resolved. The problem was in my understanding. It returns the graph with vertex "data" set to the connected components. Thanks. On Wed, Oct 22, 2014 at 6:56 PM, Manoj Awasthi wrote: > Hi Guys, > > I am trying something very basic. I am using GraphX to load a graph from > an edge list fi

Graphx connectComponents API

Hi Guys, I am trying something very basic. I am using GraphX to load a graph from an edge list file which is like this: *220 224* *400 401* *220 221* So it has following nodes (just for the sake of understanding - bear with me for drawing): *220 => 224 400 => 401 * * ||* * v 221* Cle

Which part of the code deals with communication?

Hi all, Workers will exchange data in between, right? What classes are in charge of these actions? - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org