Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/945#issuecomment-47997623
This is a good first cut for adding throttling, but I would like to extract
out the rate limiting code into a separator object, to keep the code clean.
Also, this way of throttling the rate would create a erratic data rates. A
whole lot of events will get received to very fast, and then the receiving stay
blocked for a time. The effective data rates would be something like this.
_________ __________
| | | |
___| |________| |___________
|<------ 100ms ---->|
Rather we want the rate to be much smoother. The better way to implement
this can be found in
https://github.com/apache/spark/blob/master/streaming/src/main/scala/org/apache/spark/streaming/util/RateLimitedOutputStream.scala
where the data rates (rather than # records received in batch) is checked.
This will be smoother.
So my recommendation would be to be create separate RateLimiter class which
has the logic for limiting. And the BlockGenerator would use that class.
PS: Cool talk at the summit!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
