Spark on Mesos fine-grained - has one core less per executor

I'm doing a performance analysis for Spark on Mesos and I can see that the Coarse-grained backend simply launches tasks in wave size of the amount of cores available. But it seems Fine-grained mode the Mesos executor takes 1 core for itself (so -1 core per mesos slave). Shouldn't fine- and coarse-g

Spark shell crumbles after memory is full

I'm running a query from the BigDataBenchmark, query 1B to be precise. When running this with Spark (1.3.1)+ mesos(0.21) in coarse grained mode with 5 mesos slave, through a spark shell, all is well. However rerunning the query a few times: scala> sqlContext.sql("SELECT pageURL, pageRank FROM

Scheduler delay vs. Getting result time

Hi, In the Spark UI, under “Show additional metrics”, there are two extra metrics you can show .1 Scheduler delay .2 and Getting result time When hovering “Scheduler Delay" it says (among other things): …time to send task result from executor… When hovering “Getting result time”: Time that the

Java/Spark Library for interacting with Spark API

Hi, Does anyone know of a Java/Scala library (not simply a HTTP library) for interacting with Spark through its REST/HTTP API? My “problem” is that interacting through REST induces a lot of work mapping the JSON to sensible Spark/Scala objects. So a simple example, I hope there is a library whi

Re: CREATE TABLE ignores database when using PARQUET option

I'm having the same problem, did you solve this? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/CREATE-TABLE-ignores-database-when-using-PARQUET-option-tp22824p24679.html Sent from the Apache Spark User List mailing list archive at Nabble.com.

Recreating the Mesos/Spark paper's experiments

Hi all, For my master thesis I will be characterising performance of two-level schedulers like Mesos and after reading the paper: https://www.cs.berkeley.edu/~alig/papers/mesos.pdf where Spark is also introduced I am wondering how some experiments and results came about. If this is not the plac

Re: Can't run spark-submit with an application jar on a Mesos cluster

Hi, What do the mesos slave logs say? Usually this gives a clearcut error, they are probably local on a slave node. I'm not sure about your config, so I can;t pinpoint you to a specific path. might look something like: /???/mesos/slaves/20150213-092641-84118794-5050-14978-S0/frameworks/2015032

Re: Can't run spark-submit with an application jar on a Mesos cluster

Well that are only the logs of the slaves on mesos level, I'm not sure from your reply if you can ssh into a specific slave or not, if you can, you should look at actual output of the application (spark in this case) on a slave in e.g. /tmp/mesos/slaves/20150322-040336-606645514-5050-2744-S1/fr