I believe rebalancing should be triggered by the user, using the
command "storm rebalance" or its API equivalent. The rest is about right :)
Good luck,
Ahmed
On 01/30/2015 03:46 PM, Manpreet Singh wrote:
Thanks a lot Ahmed.
It really gave me a direction. So you are saying that:
* A program should check metrics at regular intervals.
* Based on that, program can add/remove VMs to cluster. Storm will
automatically recognize the change in VMs
* Depending on VM addition or removal, program will trigger
re-balance command to change no. of worker processes or executors.
Did I get it right?
Thanks,
Manpreet.
On Thu, Jan 29, 2015 at 2:50 PM, Ahmed El Rheddane
<[email protected] <mailto:[email protected]>> wrote:
Hi all,
I have investigated ways of auto scaling Storm topologies recently
and it is perfectly feasible.
- For monitoring, you can use either the Thrift interface or the
new MetricsConsumer bolt, it's probably possible using the REST
API as well, but I haven't tried that out yet.
- For scaling, Storm automatically discovers the addition of new
nodes (VMs) to the cluster and allows you to change the number of
executors (threads) per component on the fly.
- You can as well plug in a custom scheduler, which is called upon
each rebalancing request, to place the executors of your topology
as you see fits.
I hope this could help.
Ahmed
On 01/28/2015 09:55 PM, Parth Brahmbhatt wrote:
Not sure how much progress has been made on
https://issues.apache.org/jira/browse/STORM-594 but that is
the only
initiative that I am aware of.
Thanks
Parth
On 1/28/15, 12:42 PM, "Margus Roo" <[email protected]
<mailto:[email protected]>> wrote:
I have not heard or read about it from the documentation.
Margus (margusja) Roo
http://margus.roo.ee
skype: margusja
+372 51 480
On 28/01/15 20:59, Manpreet Singh wrote:
Hi,
I am a newbie to Apache Storm and analyzing it for a
product. I am
comparing it with Gearman from parallel processing
aspect. So far
Apache Storm seems to be winning on most of the
aspects. But one of
the important aspect which I got stuck is
'auto-scale'. Auto-scale (up
as well as down) of storm resources doesn't look
possible. This is
critical for my product as stream data could vary a
lot through out
the day. Having auto-scale will save me cost as well
as monitoring the
usage every now and then.
Are there any workarounds to this problem with Apache
Storm?
Thanks in advance.
-Manpreet.
