RE: RPC Timeout and Abnormally Long JvmGcTime

2016-04-29 Thread Wes Holler
Oops... The "adasdasdasdasd" below is JSON scraped from the status API endpoint. -Original Message----- From: Wes Holler Sent: Friday, April 29, 2016 6:09 PM To: dev Subject: RPC Timeout and Abnormally Long JvmGcTime Recently we switched to EMR 4.5/Spark 1.6.1 and have since enc

RPC Timeout and Abnormally Long JvmGcTime

2016-04-29 Thread Wes Holler
Recently we switched to EMR 4.5/Spark 1.6.1 and have since encountered a new failure scenario. The primary symptom is that the cluster appears to be stalled. The job has not failed but will not proceed and has to be killed. One or more RpcTimeoutException s (see below) are usually found towards