Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/945#discussion_r14792207
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/NetworkReceiverSuite.scala
---
@@ -146,6 +146,37 @@ class NetworkReceiverSuite extends FunSuite with
Timeouts {
assert(recordedData.toSet === generatedData.toSet)
}
+ test("block generator throttling") {
+ val blockGeneratorListener = new FakeBlockGeneratorListener
+ val blockInterval = 1000
--- End diff --
It would be great if you can modify this test slightly to make it strong.
Can you try running a block interval 50 ms and a rate of 10 records / block
interval . Then send records with a sleep of 1 ms. This means you are pushing
at almost 50 records per block interval. Do this for multiple intervals, say
for a wait time of the 1000 ms. Then the assert needs to check whether *every*
block (in the `blockGeneratorListener.arrayBuffers`) has <= (maxRate *
blockInterval / 1000) records. This effectively tests that the rate is
maintained across multiple blocks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
