Make Spark Job Board permanent.

During Spark Summit 2014 there was a Job Board( http://spark-summit.org/2014/jobs) for positions related to spark technology. It is great thing, because it's hard to search for position, related to so young technology. And such board good for spark community, because it makes easy for companies to

Re: wierd caching

It might mean that some partition was computed on two nodes, because a task for it wasn't able to be scheduled locally on the first node. Did the RDD really have 426 partitions total? You can click on it and see where there are copies of each one. Matei > On Nov 8, 2014, at 10:16 PM, Nathan Kr

wierd caching

RDD NameStorage LevelCached PartitionsFraction CachedSize in MemorySize in TachyonSize on Disk 8 Memory Deserialized 1x Replicated 426 107% 59.7 GB 0.0 B 0.0 BAnyone understand what it means to have more than 100% of an rdd cached? Thanks,

Re: org/apache/commons/math3/random/RandomGenerator issue

I ran into this problem too and I know of a workaround but don't exactly know what is happening. The work around is to explicitly add either the commons math jar or your application jar (shaded with commons math) to spark.executor.extraClassPath. My hunch is that this is related to the class loade

contains in array in Spark SQL

hi, what would be the syntax for check for an attribute in an array data type for my where clause ? select * from business where cateogories contains 'X' // something like this , is this right syntax ?? attribute: categories type: Array thx srinivas

Re: Debian package for spark?

No change from 1.1.0 to 1.1.1-SNAPSHOT. The deb profile hasn't changed since before the 1.0.2 release. On Sat, Nov 8, 2014 at 3:12 PM, Kevin Burton wrote: > Weird… I’m using a 1.1.0 source tar.gz … > > but if it’s fixed in 1.1.1 that’s good. > > On Sat, Nov 8, 2014 at 2:08 PM, Mark Hamstra > w

Unresolved Attributes

I have an exception when I am trying to run a simple where clause query. I can see the name attribute is present in the schema but somehow it still throws the exception. query = "select name from business where business_id=" + business_id what am I doing wrong ? thx srinivas Exception in threa

Re: Does spark works on multicore systems?

oops, meant to cc userlist too On Sat, Nov 8, 2014 at 3:13 PM, Aaron Davidson wrote: > The default local master is "local[*]", which should use all cores on your > system. So you should be able to just do "./bin/pyspark" and > "sc.parallelize(range(1000)).count()" and see that all your cores wer

Re: Debian package for spark?

Weird… I’m using a 1.1.0 source tar.gz … but if it’s fixed in 1.1.1 that’s good. On Sat, Nov 8, 2014 at 2:08 PM, Mark Hamstra wrote: > The building of the Debian package in Spark works just fine for me -- I > just did it using a clean check-out of 1.1.1-SNAPSHOT and `mvn -U -Pdeb > -DskipTests

Does spark works on multicore systems?

I am a Spark newbie and I use python (pyspark). I am trying to run a program on a 64 core system, but no matter what I do, it always uses 1 core. It doesn't matter if I run it using "spark-submit --master local[64] run.sh" or I call x.repartition(64) in my code with an RDD, the spark program always

Do spark works on multicore systems?

I am a Spark newbie and I use python (pyspark). I am trying to run a program on a 64 core system, but no matter what I do, it always use 1 core. It doesn't matter if I run it using "spark-submit --master local[64] run.sh" or I call x.repartition(64) in my code with an RDD, the spark program always

Re: Debian package for spark?

The building of the Debian package in Spark works just fine for me -- I just did it using a clean check-out of 1.1.1-SNAPSHOT and `mvn -U -Pdeb -DskipTests clean package`. There's likely something else amiss in your build. Actually, that's not quite true. There is one small problem with the Debi

Re: org/apache/commons/math3/random/RandomGenerator issue

This means you haven't actually included commons-math3 in your application. Check the contents of your final app jar and then go check your build file again. On Sat, Nov 8, 2014 at 12:20 PM, lev wrote: > Hi, > I'm using breeze.stats.distributions.Binomial with spark 1.1.0 and having > the same er

Re: Debian package for spark?

Another note for the official debs. ‘spark’ is a bad package name because of confusion with the spark programming lang based on ada. There are packages for this already named ‘spark’ so I put mine as ‘apache-spark’ On Sat, Nov 8, 2014 at 12:21 PM, Kevin Burton wrote: > OK… here’s my version

Re: org/apache/commons/math3/random/RandomGenerator issue

Hi Lev, I also finally couldn't solve that problem and switched to Java.util.Random. Thanks~ Anny On Sat, Nov 8, 2014 at 4:21 AM, lev [via Apache Spark User List] < ml-node+s1001560n18406...@n3.nabble.com> wrote: > Hi, > I'm using breeze.stats.distributions.Binomial with spark 1.1.0 and having

Re: Debian package for spark?

OK… here’s my version. https://github.com/spinn3r/spark-deb it’s just two files really. so if the standard spark packages get fixed I’ll just switch to them. Doesn’t look like there’s an init script and the conf isn’t in /etc … On Sat, Nov 8, 2014 at 12:06 PM, Kevin Burton wrote: > looks lik

Re: Debian package for spark?

looks like it doesn’t work: > [ERROR] Failed to execute goal org.vafer:jdeb:0.11:jdeb (default) on project spark-assembly_2.10: Failed to create debian package /Users/burton/Dropbox/projects-macbook-pro-2013-09/spark-1.1.0/assembly/target/spark_1.1.0-${buildNumber}_all.deb: Could not create deb pa

Re: Debian package for spark?

Nice! Not sure how I missed that. Building it now. If it has all the init scripts and config in the right place I might use that. I might have to build a cassandra package too which adds cassandra support.. I *think* at least. Maybe distribute this .deb with the standard downloads? Kevin On

Re: org/apache/commons/math3/random/RandomGenerator issue

lev wrote > I'm using breeze.stats.distributions.Binomial with spark 1.1.0 and having > the same error. > I tried to add the dependency to math3 with versions 3.11, 3.2, 3.3 and it > didn't help. I am experiencing likewise with all the breeze.stats.distributions using any math3 version. I run 's

Debian package for spark?

Are there debian packages for spark? If not I plan on making one… I threw one together in about 20 minutes as they are somewhat easy with maven and jdeb. But of course there are other things I need to install like cassandra support and an init script. So I figured I’d ask here first. If not we

Re: Spark on YARN, ExecutorLostFailure for long running computations in map

So it seems that this problem was related to  http://apache-spark-developers-list.1001551.n3.nabble.com/Lost-executor-on-YARN-ALS-iterations-td7916.html and increasing the executor memory worked for me. __ Hi, I am getting ExecutorLost

Embedding static files in a spark app

Hi spark. I have a set of text files that are dependencies of my app. They are less than 2mb in total size. What's the idiom for packaing text file dependencies for a spark based jar file? Class resources in packages ? Or distributing them separately?

Re: org/apache/commons/math3/random/RandomGenerator issue

Hi, I'm using breeze.stats.distributions.Binomial with spark 1.1.0 and having the same error. I tried to add the dependency to math3 with versions 3.11, 3.2, 3.3 and it didn't help. Any ideas what might be the problem? Thanks, Lev. anny9699 wrote > I use the breeze.stats.distributions.Bernoulli

Re: Issue with Custom Key Class

Does your RDD contain a null key? On Sat, Nov 8, 2014 at 11:15 AM, Bahubali Jain wrote: > Hi, > I have a custom key class.In this class equals() and hashcode() have been > overridden. > I have a javaPairRDD which has this class as the key .When groupbykey() or > reducebykey() is called a null o

Issue with Custom Key Class

Hi, I have a custom key class.In this class equals() and hashcode() have been overridden. I have a javaPairRDD which has this class as the key .When groupbykey() or reducebykey() is called a null object is being passed to the function *equals*(Object obj) as a result the grouping is failing. Is t

Spark on YARN, ExecutorLostFailure for long running computations in map

Hi, I am getting ExecutorLostFailure when I run spark on YARN and in map I perform very long tasks (couple of hours). Error Log is below. Do you know if it is possible to set something to make it possible for Spark to perform these very long running jobs in map? Thank you very much for any adv