Thanks, Rajesh. I think that acquring/relinquishing executors is important but I feel like there are at least two layers for resource allocation and autoscaling. It seems that acquiring and relinquishing executors is a way to optimize resource utilization within a pre-set Spark cluster of machines.
However, to accommodate for big spikes in input data, we also need the actual cluster scaling, i.e. adding (or removing, when no longer needed) worker node machines automatically. On that front, I wonder how Spark reacts to a machine being added or removed and what the actual procedure would be. If we're running on a Hadoop cluster, there's a description of adding a node <http://wiki.apache.org/hadoop/FAQ#I_have_a_new_node_I_want_to_add_to_a_running_Hadoop_cluster.3B_how_do_I_start_services_on_just_one_node.3F> there. There's also discussions of Hadoop node adding/removal such as this one <http://stackoverflow.com/questions/16774439/how-do-i-correctly-remove-nodes-in-hadoop> . My worry is, will Spark "gracefully" and "quickly" detect the presence of a new node and start utilizing it (i.e. how much does it communicate with the Hadoop cluster manager?)... By the same token, if a node is removed, how can it be removed gracefully so as not to affect/kill any running Spark jobs? On Wed, May 27, 2015 at 10:57 PM, <rajesh_kall...@dellteam.com> wrote: > *Dell - Internal Use - Confidential * > > Did you check > https://drive.google.com/file/d/0B7tmGAdbfMI2OXl6azYySk5iTGM/edit and > http://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation > > > > Not sure if the spark kafka receiver emits metrics on the lag, check this > link out > http://community.spiceworks.com/how_to/77610-how-far-behind-is-your-kafka-consumer > > > > You should be able to whip up a script that runs the Kafka > ConsumerOffsetChecker periodically and pipe it to a metrics backend of your > choice. Based on this you can work the dynamic resource allocation magic. > > -----Original Message----- > From: dgoldenberg [mailto:dgoldenberg...@gmail.com] > Sent: Wednesday, May 27, 2015 6:21 PM > To: user@spark.apache.org > Subject: Autoscaling Spark cluster based on topic sizes/rate of growth in > Kafka or Spark's metrics? > > Hi, > > I'm trying to understand if there are design patterns for autoscaling > Spark (add/remove slave machines to the cluster) based on the throughput. > > Assuming we can throttle Spark consumers, the respective Kafka topics we > stream data from would start growing. What are some of the ways to generate > the metrics on the number of new messages and the rate they are piling up? > This perhaps is more of a Kafka question; I see a pretty sparse javadoc > with the Metric interface and not much else... > > What are some of the ways to expand/contract the Spark cluster? Someone > has mentioned Mesos... > > I see some info on Spark metrics in the Spark monitoring guide . Do we > want to perhaps implement a custom sink that would help us autoscale up or > down based on the throughput? > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Autoscaling-Spark-cluster-based-on-topic-sizes-rate-of-growth-in-Kafka-or-Spark-s-metrics-tp23062.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional > commands, e-mail: user-h...@spark.apache.org >
