This is not supposed to happen. Do you have a repro?
On Tue, Dec 6, 2016 at 6:11 PM, Nicholas Chammas wrote:
> [Re-titling thread.]
>
> OK, I see that the exception from my original email is being triggered
> from this part of UnsafeInMemorySorter:
>
> https://github.com/apache/spark/blob/v2.0.
[Re-titling thread.]
OK, I see that the exception from my original email is being triggered from
this part of UnsafeInMemorySorter:
https://github.com/apache/spark/blob/v2.0.2/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java#L209-L212
So I can ask a more
This sounds much better.
Follow up question is if we should provide MAP@k, which I believe is
wider used metric.
On 12/06/2016 09:52 PM, Sean Owen wrote:
> As I understand, this might best be called "mean precision@k", not
> "mean average precision, up to k".
>
> On Tue, Dec 6, 2016 at 9:43 PM M
Hi Jakob,
It seems like I’ll have to either replace the version with my custom version in
all the pom.xml files in every subdirectory that has one and publish locally,
or keep the version (i.e. 2.0.2) and manually remove the spark repository cache
in ~/.ivy2 and ~/.m2 and publish spark locally
As I understand, this might best be called "mean precision@k", not "mean
average precision, up to k".
On Tue, Dec 6, 2016 at 9:43 PM Maciej Szymkiewicz
wrote:
> Thank you Sean.
>
> Maybe I am just confused about the language. When I read that it returns "the
> average precision at the first k ra
Hi
If there are some way to see the bytecode in each task that is executed by
spark.
Thanks
-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
Client mode or cluster mode?
On Mon, Dec 5, 2016 at 10:05 PM, Yu Wei wrote:
> Hi Guys,
>
>
> I tried to run spark on mesos cluster.
>
> However, when I tried to submit jobs via spark-submit. The driver is in
> "Queued state" and not started.
>
>
> Which should I check?
>
>
>
> Thanks,
>
> Jared,
I tried one example on sparkR:
> training <- suppressWarnings(createDataFrame(iris))> step(spark.glm(training, Sepal_Width ~ Sepal_Length + Species), direction = "forward")
There is an error:
Error: $ operator not defined for this S4 class
Based on my understanding of mllib.R, I think it is n
Well other than making the code consistent whats the high level goal in doing
this and why does it matter so much how many workers we have in different
scenarios (pyspark versus different components of spark). I'm ok not making
the change and working on something else to be honest but spending
Yes, I think changing the property (line 29) in spark's root
pom.xml should be sufficient. However, keep in mind that you'll also
need to publish spark locally before you can access it in your test
application.
On Tue, Dec 6, 2016 at 2:50 AM, Teng Long wrote:
> Thank you Jokob for clearing thing
Steve,
I appreciate your experience and insight when dealing with large clusters at
the data-center scale. I'm also well aware of the complex nature of schedulers,
and that it is an area of ongoing research being done by people/companies with
many more resources than I have. This might explain
Thank you Sean.
Maybe I am just confused about the language. When I read that it returns
"the average precision at the first k ranking positions" I somehow
expect there will ap@k there and a the final output would be MAP@k not
average precision at the k-th position.
I guess it is not enough sleep
Ah cool, thanks for the link!
On 6 December 2016 at 12:25, Nick Pentreath
wrote:
> Indeed, it's being tracked here: https://issues.apache.
> org/jira/browse/SPARK-18230 though no Pr has been opened yet.
>
>
> On Tue, 6 Dec 2016 at 13:36 chris snow wrote:
>
>> I'm using the MatrixFactorizationMo
Indeed, it's being tracked here:
https://issues.apache.org/jira/browse/SPARK-18230 though no Pr has been
opened yet.
On Tue, 6 Dec 2016 at 13:36 chris snow wrote:
> I'm using the MatrixFactorizationModel.predict() method and encountered
> the following exception:
>
> Name: java.util.NoSuchElemen
Nicholas,
FYI, there's some patch for Hadoop 2.8? 2.9? to move up to Netty
https://issues.apache.org/jira/browse/HADOOP-13866
https://issues.apache.org/jira/browse/HADOOP-12854
On 5 Dec 2016, at 19:46, Nicholas Chammas
mailto:nicholas.cham...@gmail.com>> wrote:
So if I'm running Spark 2.0.2
This is essentially what the cluster schedulers do: allow different people to
submit work with different credentials and priority; cgroups & equivalent to
limit granted resources to requested ones. If you have pre-emption enabled, you
can even have one job kill work off the others. Spark does re
I'm using the MatrixFactorizationModel.predict() method and encountered the
following exception:
Name: java.util.NoSuchElementException
Message: next on empty iterator
StackTrace: scala.collection.Iterator$$anon$2.next(Iterator.scala:39)
scala.collection.Iterator$$anon$2.next(Iterator.scala:37)
sc
jenkins uses SBT, so you need to do the test run there. They are different, and
have different test runners in particular.
On 30 Nov 2016, at 04:14, Saikat Kanjilal
mailto:sxk1...@hotmail.com>> wrote:
Hello Spark dev community,
I took this the following jira item
(https://github.com/apache/sp
Thank you Jokob for clearing things up for me.
Before, I thought my application was compiled against my local build since I
can get all the logs I just added in spark-core. But it was all along using
spark downloaded from remote maven repository, and that’s why I “cannot" add
new RDD methods i
19 matches
Mail list logo
