Got it, thank you, Tathagata and Ted. Could you comment on my other question <http://apache-spark-user-list.1001560.n3.nabble.com/Autoscaling-Spark-cluster-based-on-topic-sizes-rate-of-growth-in-Kafka-or-Spark-s-metrics-tt23062.html> as well? Basically, I'm trying to get a handle on a good approach to throttling, on the one hand side, and autoscaling the cluster, on the other. Are there any recommended approaches or design patterns for autoscaling that you have implemented or could point me at? Thanks!
On Wed, May 27, 2015 at 8:08 PM, Tathagata Das <t...@databricks.com> wrote: > You can throttle the no receiver direct Kafka stream using > spark.streaming.kafka.maxRatePerPartition > <http://spark.apache.org/docs/latest/configuration.html#spark-streaming> > > > On Wed, May 27, 2015 at 4:34 PM, Ted Yu <yuzhih...@gmail.com> wrote: > >> Have you seen >> http://stackoverflow.com/questions/29051579/pausing-throttling-spark-spark-streaming-application >> ? >> >> Cheers >> >> On Wed, May 27, 2015 at 4:11 PM, dgoldenberg <dgoldenberg...@gmail.com> >> wrote: >> >>> Hi, >>> >>> With the no receivers approach to streaming from Kafka, is there a way to >>> set something like spark.streaming.receiver.maxRate so as not to >>> overwhelm >>> the Spark consumers? >>> >>> What would be some of the ways to throttle the streamed messages so that >>> the >>> consumers don't run out of memory? >>> >>> >>> >>> >>> >>> -- >>> View this message in context: >>> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-from-Kafka-no-receivers-and-spark-streaming-receiver-maxRate-tp23061.html >>> Sent from the Apache Spark User List mailing list archive at Nabble.com. >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>> For additional commands, e-mail: user-h...@spark.apache.org >>> >>> >> >
