Thanks a lot Ahmed. It really gave me a direction. So you are saying that:
- A program should check metrics at regular intervals. - Based on that, program can add/remove VMs to cluster. Storm will automatically recognize the change in VMs - Depending on VM addition or removal, program will trigger re-balance command to change no. of worker processes or executors. Did I get it right? Thanks, Manpreet. On Thu, Jan 29, 2015 at 2:50 PM, Ahmed El Rheddane < [email protected]> wrote: > Hi all, > > I have investigated ways of auto scaling Storm topologies recently and it > is perfectly feasible. > - For monitoring, you can use either the Thrift interface or the new > MetricsConsumer bolt, it's probably possible using the REST API as well, > but I haven't tried that out yet. > - For scaling, Storm automatically discovers the addition of new nodes > (VMs) to the cluster and allows you to change the number of executors > (threads) per component on the fly. > - You can as well plug in a custom scheduler, which is called upon each > rebalancing request, to place the executors of your topology as you see > fits. > > I hope this could help. > > Ahmed > > > On 01/28/2015 09:55 PM, Parth Brahmbhatt wrote: > >> Not sure how much progress has been made on >> https://issues.apache.org/jira/browse/STORM-594 but that is the only >> initiative that I am aware of. >> >> Thanks >> Parth >> >> On 1/28/15, 12:42 PM, "Margus Roo" <[email protected]> wrote: >> >> I have not heard or read about it from the documentation. >>> >>> Margus (margusja) Roo >>> http://margus.roo.ee >>> skype: margusja >>> +372 51 480 >>> >>> On 28/01/15 20:59, Manpreet Singh wrote: >>> >>>> Hi, >>>> >>>> I am a newbie to Apache Storm and analyzing it for a product. I am >>>> comparing it with Gearman from parallel processing aspect. So far >>>> Apache Storm seems to be winning on most of the aspects. But one of >>>> the important aspect which I got stuck is 'auto-scale'. Auto-scale (up >>>> as well as down) of storm resources doesn't look possible. This is >>>> critical for my product as stream data could vary a lot through out >>>> the day. Having auto-scale will save me cost as well as monitoring the >>>> usage every now and then. >>>> >>>> Are there any workarounds to this problem with Apache Storm? >>>> >>>> Thanks in advance. >>>> >>>> -Manpreet. >>>> >>> >
