Hazem Mahmoud created SPARK-21099:
-------------------------------------
Summary: INFO Log Message Using Incorrect Executor Idle Timeout
Value
Key: SPARK-21099
URL: https://issues.apache.org/jira/browse/SPARK-21099
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 2.1.0, 1.6.0
Reporter: Hazem Mahmoud
INFO log message is using the wrong idle timeout
(spark.dynamicAllocation.executorIdleTimeout) when printing the message that
the executor holding the RDD cache is being removed.
INFO spark.ExecutorAllocationManager: Removing executor 1 because it has been
idle for 30 seconds (new desired total will be 0)
It should be using spark.dynamicAllocation.cachedExecutorIdleTimeout when the
RDD cache timeout is reached. I was able to confirm this by doing the following:
1. Update spark-defaults.conf to set the following:
executorIdleTimeout=30
cachedExecutorIdleTimeout=20
2. Update log4j.properties to set the following:
shell.log.level=INFO
3. Run the following in spark-shell:
scala> val textFile = sc.textFile("/user/spark/applicationHistory/app_1234")
scala> textFile.cache().count()
4. After 30 secs you will see 2 timeout messages, but of which are 30 secs
(whereas one *should* be for 20 secs)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
