: Thursday, April 23, 2015 4:46 PM
To: Andrew Leverentz
Cc: user@spark.apache.org
Subject: Re: Understanding Spark/MLlib failures
Hi Andrew,
I observed similar behavior under high GC pressure, when running ALS. What
happened to me was that, there would be very long Full GC pauses (over 600
seconds
: Understanding Spark/MLlib failures
Hi Andrew,
The .principalComponents feature of RowMatrix is currently constrained to tall
and skinny matrices. Your matrix is barely above the skinny requirement (10k
columns), though the number of rows is fine.
What are you looking to do with the principal
Hi Andrew, according to you we should balance the time when gc run and the
batch time, which rdd is processed?
On Fri, Apr 24, 2015 at 6:58 AM Reza Zadeh wrote:
> Hi Andrew,
>
> The .principalComponents feature of RowMatrix is currently constrained to
> tall and skinny matrices. Your matrix is b
Hi Andrew,
The .principalComponents feature of RowMatrix is currently constrained to
tall and skinny matrices. Your matrix is barely above the skinny
requirement (10k columns), though the number of rows is fine.
What are you looking to do with the principal components? If unnormalized
PCA is OK f
Hi Andrew,
I observed similar behavior under high GC pressure, when running ALS. What
happened to me was that, there would be very long Full GC pauses (over 600
seconds at times). These would prevent the executors from sending
heartbeats to the driver. Then the driver would think that the executor
