Myself and others on here encounter situations where tasks get stuck on the
last stage and the entire job just hangs. No exceptions or nothing. Most likely
a deadlock somewhere inside spark.
Sent from my Verizon Wireless 4G LTE smartphone
-------- Original message --------
From: Andy Max <[email protected]>
Date: 02/11/2016 2:44 PM (GMT-05:00)
To: Darren Govoni <[email protected]>
Cc: [email protected]
Subject: Re: Spark workers disconnecting on 1.5.2
No, ours are running on Docker containers spread across few physical servers.
Databricks runs their service on AWS. Wonder if they are seeing this issues.
What other problems are you seeing with 1.5.2? We are seeing some executors
get stuck in loading state.
On Thu, Feb 11, 2016 at 11:16 AM, Darren Govoni <[email protected]> wrote:
I see this too. Might explain some other serious problems we're having with
1.5.2
Is your cluster in AWS?
Sent from my Verizon Wireless 4G LTE smartphone
-------- Original message --------
From: Andy Max <[email protected]>
Date: 02/11/2016 2:12 PM (GMT-05:00)
To: [email protected]
Subject: Spark workers disconnecting on 1.5.2
I launched a 4 node Spark 1.5.2 cluster. No activity for a day or so. Now
noticed that few of the workers are disconnected.
Don't see this issue on Spark 1.4 or Spark 1.3.
Would appreciate any pointers.
Thx
