Tony Novak created SPARK-19114:
----------------------------------
Summary: Backpressure rate is cast from double to long to double
Key: SPARK-19114
URL: https://issues.apache.org/jira/browse/SPARK-19114
Project: Spark
Issue Type: Bug
Reporter: Tony Novak
We have a Spark streaming job where each record takes well over a second to
execute, so the stable rate is under 1 element/second. We set
spark.streaming.backpressure.enabled=true and
spark.streaming.backpressure.pid.minRate=0.1, but backpressure did not appear
to be effective, even though the TRACE level logs from PIDRateEstimator showed
that the new rate was 0.1.
As it turns out, even though the minRate parameter is a Double, and the rate
estimate generated by PIDRateEstimator is a Double as well, RateController
casts the new rate to a Long. As a result, if the computed rate is less than 1,
it's truncated to 0, which ends up being interpreted as "no limit".
What's particularly confusing is that the Guava RateLimiter class takes a rate
limit as a double, so the long value ends up being cast back to a double.
Is there any reason not to keep the rate limit as a double all the way through?
I'm happy to create a pull request if this makes sense.
We encountered the bug on Spark 1.6.2, but it looks like the code in the master
branch is still affected.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
