We are seeing lots of stability problems with Spark 2.1.1 as a result of
dropped events. We disabled the event log, which seemed to help, but many
events are still being dropped, as in the example log below.
I there any way for me to see what listener is backing up the queue? Is
there any workaro
ted.
>
> I might be mistaken, if somebody has a good explanation, would also like
> to hear.
>
> On Fri, May 19, 2017 at 5:45 PM, Miles Crawford
> wrote:
>
>> Hey ya'll,
>>
>> Trying to migrate from Spark 1.6.1 to 2.1.0.
>>
>> I use EMR, and
Could I be experiencing the same thing?
https://www.dropbox.com/s/egtj1056qeudswj/sparkwut.png?dl=0
On Wed, Nov 16, 2016 at 10:37 AM, Shreya Agarwal
wrote:
> I think that is a bug. I have seen that a lot especially with long running
> jobs where Spark skips a lot of stages because it has pre-co
Hey ya'll,
Trying to migrate from Spark 1.6.1 to 2.1.0.
I use EMR, and launched a new cluster using EMR 5.5, which runs spark 2.1.0.
I updated my dependencies, and fixed a few API changes related to
accumulators, and presto! my application was running on the new cluster.
But the application UI
Hello folks. I recently migrated my application to Spark 2.0, and
everything worked well, except for one function that uses "toDS" and the ML
libraries.
This stage used to complete in 15 minutes or so on 1.6.2, and now takes
almost two hours.
The UI shows very strange behavior - completed stages
It is completed apps that are not showing up. I'm fine with incomplete apps
not appearing.
On Tue, Apr 12, 2016 at 6:43 AM, Steve Loughran
wrote:
>
> On 12 Apr 2016, at 00:21, Miles Crawford wrote:
>
> Hey there. I have my spark applications set up to write their event logs
&g
Hey there. I have my spark applications set up to write their event logs
into S3 - this is super useful for ephemeral clusters, I can have
persistent history even though my hosts go away.
A history server is set up to view this s3 location, and that works fine
too - at least on startup.
The probl
