On Jan 19, 2009, at 10:07 AM, Larry Ludwig wrote:
>
> Brian Gupta wrote:
>>
>> Mostly we want the ability, to leverage the micro-accounting EC2
>> offers us. (To lower operational costs).
>>
>> Since EC2 bills by the hour, wouldn't it be prudent to be able to
>> spin
>> down idling webservers during the evening, and spin up extra ones
>> when
>> you know there are upcoming high traffic website events? Currently we
>> have this ability. For now it is still a manual process, in that we
>> have to say "please spin up X webserver nodes now in the xyz
>> environment", but they do autoconfigure themselves, update DNS, and
>> even adjust the load balancer config. We are still working on the
>> scheduling ability, but our puppet code and ec2 glue are all
>> complete,
>> if should be fairly straightforward to throw something together. (We
>> might even leverage the OS scheduler.)
>>
> Hmm this brings up an interesting discussion.
>
> It depends... if a no-op CPU command is billable to EC2 and they are
> truly sitting idle does Amazon charge for it? Obliviously a running
> instance will have CPU cycles, but not much if idle.
>
What the CPU is doing isn't really relevant, the EC2 machines are
really just Xen instances, and you pay for every hour (or portion of
an hour) that the instance is booted, regardless of whether it is
working or not.
> Also the build process of a new instance I suspect has a decent amount
> of CPU time (obviously less if you are using binaries)
>
Not really, these are more like copy-on-write clones of an existing
filesystem, so you don't really have to build a new instance, you just
boot an instance from a filesystem that already exists.
> I'm sure there is some point where it makes more sense to leave the
> instance running for X amount of hours, instead of regening it.
> (meaning
> it costs more to build a new EC2 instance, than just leave an existing
> one active) The other issues related to re-gening is new IP address
> and
> the time for EC2 to have your new instance ready (ie 10 min)
>
It doesn't cost any more to start a new instance than it does to leave
it running, you still just pay by the hour. What would make sense
though is to figure out how long the instance has been running before
deciding whether to shut it down. For example, if I have an instance
that is idle but has been running for 6 hours and 5 minutes, I'm going
to pay for that seventh hour whether I shut the instance down now or
in 55 minutes, so it makes sense to wait another 45 minutes or so to
see if some more work does come up. The other part of this is that if
you shut them down too aggressively, you may end up paying more, such
as if you shut an instance down after only 1 minute of idle time, you
may find that you had 4 instances start up, do 5 minutes worth of
work, and shut down only a few minutes apart. In that case you paid
for 4 hours worth of EC2 time, but only got 20 minutes worth of work
out of it.
In many applications, the new IP address is not an issue. For a web
farm for example, I'll usually run something like Varnish on the front-
end machines with the public IP addresses, and keep them running all
the time, then bring up or shut down backend hosts behind varnish as
needed, and have an init script that registers the new backends as
being available as part of their boot up.
> Have you done any investigation on the cost benefits? If so what were
> the results?
>
> I'm obviously all for the automating the build of a new instance, and
> also automatically adding it to the pool of resources. I'm not so
> sure
> the next level of automation. Auto-scaling without any human
> interaction.
>
> The issue I have with most auto-scaling is the very basic metrics used
> to measure when to add/remove resources. I believe the issues are
> very customer/app specific and while with one customer X metric
> might be
> valid, with another you need X, Y and Z in some formula to determine
> when to go to the next level. The other issue related to this is the
> application itself. What may have allowed a customer app to scale to
> one level, may require code changes to scale effectively at a much
> higher level. This is something auto-scaling could never do.
> Vertical
> scaling is ALWAYS easier than going horizontal.
>
> Puppet is great at the automation of administration. To me auto-
> scaling
> could be a different tool and write high level rules on when to scale
> and what part to scale. Do I hear another tool in the making? :-)
>
I've got several auto-scaling tools that I've written for different
applications in the past, now that I find myself abruptly unemployed
maybe it's time to package them up into something more generically
useful. :)
>> Our eventual goal has always been *being able* to autoscale.
>> Realistically though, since unexpected load is fairly uncommon, we
>> still want a human in the scaling feedback loop. (So basically we
>> want
>> autoscaling with a "Human! Press this button!" step in the
>> autoscaling
>> process.)
>>
I usually go with an "automated within reason" definition, something
along the lines of:
when ( load is greater X for Y minutes ) {
total=count_total_number_of_running_instances()
recent=count_number_of_instances_started_in_the_last_hour()
if ( total < MAX_TOTAL AND recent < MAX_RECENT ) {
start_a_new_instance()
} else {
notify_administrator()
}
}
--
Jason Kohles, RHCA RHCDS RHCE
em...@jasonkohles.com - http://www.jasonkohles.com/
"A witty saying proves nothing." -- Voltaire
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---
