Hi Aravindh, spark.streaming.receiver.maxRate is per receiver. You should multiply with number of receivers with max rate.
Regards, Sai On Mon, Sep 19, 2016 at 9:31 AM, Aravindh [via Apache Spark User List] < ml-node+s1001560n27754...@n3.nabble.com> wrote: > I am trying to throttle my spark cluster reading events from kinesis by > setting spark.streaming.receiver.maxRate to 100. The batch interval is 1 > second. But spark seems to read thousands of events for every second. Am I > missing anything here? > > ------------------------------ > If you reply to this email, your message will be added to the discussion > below: > http://apache-spark-user-list.1001560.n3.nabble.com/Kinesis- > Receiver-not-respecting-spark-streaming-receiver-maxRate-tp27754.html > To start a new topic under Apache Spark User List, email > ml-node+s1001560n1...@n3.nabble.com > To unsubscribe from Apache Spark User List, click here > <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=dG9zYWlnYW5lc2hAZ21haWwuY29tfDF8LTExOTE1Nzk1MTU=> > . > NAML > <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > ----- Sai Ganesh -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Kinesis-Receiver-not-respecting-spark-streaming-receiver-maxRate-tp27754p27758.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
