Re: [lopsa-tech] Monitoring your AWS instances

2014-10-13 Thread Matthew Barr
On Oct 10, 2014, at 10:12 PM, Edward Ned Harvey (lopser) wrote: >> From: tech-boun...@lists.lopsa.org [mailto:tech-boun...@lists.lopsa.org] >> On Behalf Of Simon Lyall >> >> Any reason you are running a t1.micro instead of a t2.micro ? The t2's are >> cheaper (especially in US regions), have mo

Re: [lopsa-tech] Monitoring your AWS instances

2014-10-13 Thread Edward Ned Harvey (lopser)
> From: tech-boun...@lists.lopsa.org [mailto:tech-boun...@lists.lopsa.org] > On Behalf Of Yves Dorfsman > > 3) If you use AWS as a cloud infrastructure, then surely, you've automated > the > installs etc... You should be able to move to a different provider in a day or > two. To be honest, you're

Re: [lopsa-tech] Monitoring your AWS instances

2014-10-11 Thread Yves Dorfsman
On 2014-10-11 07:29, Edward Ned Harvey (lopser) wrote: > > For the time being, I'm strongly in favor of Digital Ocean. The one server I > was able to easily migrate away from AWS to DO, is all-SSD, and depending on > what you measure, performs between 4x and 1000x faster in every way, and > co

Re: [lopsa-tech] Monitoring your AWS instances

2014-10-11 Thread Yves Dorfsman
On 2014-10-11 06:51, Edward Ned Harvey (lopser) wrote: > Like I said about it not being easy to uproot your server and *either* > upgrade to bigger hardware on their service, or migrating all your services > to another provider. Three things: 1) If you been using a t1.small "for years", you've

Re: [lopsa-tech] Monitoring your AWS instances

2014-10-11 Thread Edward Ned Harvey (lopser)
> From: Brandon Allbery [mailto:allber...@gmail.com] > > On Sat, Oct 11, 2014 at 8:51 AM, Edward Ned Harvey (lopser) > wrote: > It wasn't like that at the time I bought into it and committed to it.  It > degraded > into that. > > It happens. True. And hence, a conversation that started with m

Re: [lopsa-tech] Monitoring your AWS instances

2014-10-11 Thread Edward Ned Harvey (lopser)
> From: tech-boun...@lists.lopsa.org [mailto:tech-boun...@lists.lopsa.org] > On Behalf Of Matt Lawrence > > > Now, if it was basic electrical power, maybe that would be true. The power > > company certainly shouldn't be offering you a discount on your electric > > bill, with the caveat that they c

Re: [lopsa-tech] Monitoring your AWS instances

2014-10-11 Thread Edward Ned Harvey (lopser)
> From: tech-boun...@lists.lopsa.org [mailto:tech-boun...@lists.lopsa.org] > On Behalf Of Josh Smift > > which leads me to > speculate that other than memory, they may be pretty equivalent in terms > of their network performance and reliability. Responding by quoting myself: > On the "Instance T

Re: [lopsa-tech] Monitoring your AWS instances

2014-10-11 Thread Brandon Allbery
On Sat, Oct 11, 2014 at 8:51 AM, Edward Ned Harvey (lopser) < lop...@nedharvey.com> wrote: > It wasn't like that at the time I bought into it and committed to it. It > degraded into that. It happens. Providers are fickle; "caveat emptor" applies. Elsewhere I was just reminded that when POSIX re

Re: [lopsa-tech] Monitoring your AWS instances

2014-10-11 Thread Edward Ned Harvey (lopser)
> From: tech-boun...@lists.lopsa.org [mailto:tech-boun...@lists.lopsa.org] > On Behalf Of Josh Smift > > ENH> They shouldn't allow the smaller instances to degrade to the point of > ENH> being unusable, literally triggering alerts that ping failed for 3 > ENH> minutes, and serving blank white page

Re: [lopsa-tech] Monitoring your AWS instances

2014-10-10 Thread Matt Lawrence
On Fri, 10 Oct 2014, Josh Smift wrote: Now, if it was basic electrical power, maybe that would be true. The power company certainly shouldn't be offering you a discount on your electric bill, with the caveat that they can turn off your power whenever other higher-paying customers need more. Righ

Re: [lopsa-tech] Monitoring your AWS instances

2014-10-10 Thread Josh Smift
I deleted the thread before remembering another thing I wanted to mention, so I can't quote people who were saying otherwise, but: Are "micro" instances all that much less reliable than other similar things? https://aws.amazon.com/ec2/pricing/ (lightly edited for compactness) shows vC

Re: [lopsa-tech] Monitoring your AWS instances

2014-10-10 Thread Josh Smift
ENH> They shouldn't allow the smaller instances to degrade to the point of ENH> being unusable, literally triggering alerts that ping failed for 3 ENH> minutes, and serving blank white pages to web clients and performing ENH> on-par with 1998 dial up modems, and taking 10 minutes to start apache.

Re: [lopsa-tech] Monitoring your AWS instances

2014-10-10 Thread Edward Ned Harvey (lopser)
> From: tech-boun...@lists.lopsa.org [mailto:tech-boun...@lists.lopsa.org] > On Behalf Of Simon Lyall > > Any reason you are running a t1.micro instead of a t2.micro ? The t2's are > cheaper (especially in US regions), have more RAM and CPU etc. Didn't know about t2 until today. And didn't consi

Re: [lopsa-tech] Monitoring your AWS instances

2014-10-10 Thread Simon Lyall
On Sat, 11 Oct 2014, Edward Ned Harvey (lopser) wrote: 236M of memory free out of 608M. Means it's not experiencing memory pressure. Any reason you are running a t1.micro instead of a t2.micro ? The t2's are cheaper (especially in US regions), have more RAM and CPU etc. The t2 CPU credit li

Re: [lopsa-tech] Monitoring your AWS instances

2014-10-10 Thread Edward Ned Harvey (lopser)
> From: tech-boun...@lists.lopsa.org [mailto:tech-boun...@lists.lopsa.org] > On Behalf Of Morgan Blackthorne > > As a tiny dev/test box I think micros are probably fine, but you should expect > then to perform for crap. That's what they're designed to be... cheap, > disposable test instances. Anyt

Re: [lopsa-tech] Monitoring your AWS instances

2014-10-10 Thread Yves Dorfsman
On 2014-10-10 07:51, Page, Jeremy wrote: > On 10/10/2014 08:50 AM, Brandon Allbery wrote: >> On Fri, Oct 10, 2014 at 6:33 AM, Edward Ned Harvey (lopser) >> mailto:lop...@nedharvey.com>> wrote: >> >> taking 10 minutes to start apache >> >> >> Why are you running Apache on a micro instance? That

Re: [lopsa-tech] Monitoring your AWS instances

2014-10-10 Thread Morgan Blackthorne
As a tiny dev/test box I think micros are probably fine, but you should expect then to perform for crap. That's what they're designed to be... cheap, disposable test instances. Anything used by more than one person likely shouldn't be a micro instance, ever. -- ~*~ StormeRider ~*~ "Every world ne

Re: [lopsa-tech] Monitoring your AWS instances

2014-10-10 Thread Brandon Allbery
On Fri, Oct 10, 2014 at 9:51 AM, Page, Jeremy wrote: > I use micro to test chef scripts. Apache (assuming you mean httpd) works > fine as a static httpd. Or am I missing the sarcasm tag? Last I checked, micro instances weren't really provisioned for even a bare bones Apache httpd. nginx, maybe

Re: [lopsa-tech] Monitoring your AWS instances

2014-10-10 Thread Page, Jeremy
On 10/10/2014 08:50 AM, Brandon Allbery wrote: On Fri, Oct 10, 2014 at 6:33 AM, Edward Ned Harvey (lopser) mailto:lop...@nedharvey.com>> wrote: taking 10 minutes to start apache Why are you running Apache on a micro instance? That's the equivalent of running it on an original IBM PC. I use mic

Re: [lopsa-tech] Monitoring your AWS instances

2014-10-10 Thread Brandon Allbery
On Fri, Oct 10, 2014 at 6:33 AM, Edward Ned Harvey (lopser) < lop...@nedharvey.com> wrote: > taking 10 minutes to start apache Why are you running Apache on a micro instance? That's the equivalent of running it on an original IBM PC. -- brandon s allbery kf8nh sin

Re: [lopsa-tech] Monitoring your AWS instances

2014-10-10 Thread Edward Ned Harvey (lopser)
> From: tech-boun...@lists.lopsa.org [mailto:tech-boun...@lists.lopsa.org] > On Behalf Of Edward Ned Harvey (lopser) > > modems, and taking 10 minutes to start apache. That *is* a big F-U, or just a > shitty service. FWIW, the service degradation to the point of being merely shitty service, has

Re: [lopsa-tech] Monitoring your AWS instances

2014-10-10 Thread Edward Ned Harvey (lopser)
> From: tech-boun...@lists.lopsa.org [mailto:tech-boun...@lists.lopsa.org] > On Behalf Of Josh Smift > > I've only sort of been skimming this, but it doesn't seem to me like > there's anything shady or problematic about a company offering to sell you > lower-quality resources for a lower price. It

Re: [lopsa-tech] Monitoring your AWS instances

2014-10-09 Thread Yves Dorfsman
On 2014-10-09 04:35, Edward Ned Harvey (lopser) wrote: >> From: tech-boun...@lists.lopsa.org [mailto:tech-boun...@lists.lopsa.org] >> On Behalf Of Yves Dorfsman >> >> There are a lot of things on AWS that aren't clear, but this isn't one of >> them. The only conditions, beside technical issues, whe

Re: [lopsa-tech] Monitoring your AWS instances

2014-10-09 Thread Josh Smift
I've only sort of been skimming this, but it doesn't seem to me like there's anything shady or problematic about a company offering to sell you lower-quality resources for a lower price. It's not "a big fuck you to small customers", it's you getting what you're paying for. Especially if you're actu

Re: [lopsa-tech] Monitoring your AWS instances

2014-10-09 Thread Edward Ned Harvey (lopser)
> From: tech-boun...@lists.lopsa.org [mailto:tech-boun...@lists.lopsa.org] > On Behalf Of Yves Dorfsman > > There are a lot of things on AWS that aren't clear, but this isn't one of > them. The only conditions, beside technical issues, where AWS will slow > down > or shutdown your instances, regar

Re: [lopsa-tech] Monitoring your AWS instances

2014-10-08 Thread Yves Dorfsman
On 2014-10-08 19:38, Edward Ned Harvey (lopser) wrote: > > After speaking to a whole bunch of people about a whole bunch of systems, I'm > pretty confident that we know what's going on - Simply, the machine in > question is a "tiny" instance, so when Amazon gets enough demand for higher > payin

Re: [lopsa-tech] Monitoring your AWS instances

2014-10-08 Thread Morgan Blackthorne
Ah, yeah. Networking performance is rated to be bad on the micro instances. You want at least a medium for anything serious, if only for network performance. As for Zabbix vs Nagios, I think they're both good at different jobs. Zabbix is great at visualizing and analyzing metrics; if you want to b

Re: [lopsa-tech] Monitoring your AWS instances

2014-10-08 Thread Edward Ned Harvey (lopser)
> From: tech-boun...@lists.lopsa.org [mailto:tech-boun...@lists.lopsa.org] > On Behalf Of Lawrence K. Chen, P.Eng. > > My first thought, run nagios somewhere...because I have a nagios server at Are you saying you think nagios is better than zabbix? I mentioned this is the zabbix server (maybe y

Re: [lopsa-tech] Monitoring your AWS instances

2014-10-08 Thread Morgan Blackthorne
VA East has had problems in the past; it's their original datacenter so it has the oldest hardware. That said, the last issue I ran into was my VPC NAT instance (in us-west-2 / Oregon) suddenly deciding to go offline. Instance was still running, but flagged as nonresponsive. I was planning on repla

Re: [lopsa-tech] Monitoring your AWS instances

2014-10-08 Thread Lawrence K. Chen, P.Eng.
Speaking of monitoring AWS My first thought, run nagios somewhere...because I have a nagios server at home, and in addition to monitor all sorts of things at home...I have it monitoring some of the websites I host on the outside. Namely with Dreamhost and 1&1. Right now, nagios is repor

Re: [lopsa-tech] Monitoring your AWS instances

2014-10-01 Thread Edward Ned Harvey (lopser)
> From: Sean Lally [mailto:sean.la...@crownpeak.com] > > Interesting, maybe the issue was the zabbix server?  If you set it up in a vpc > you can hang on to both public and private ip... Being a non-vpc, stopping the instance causes your private IP to change, and causes your elastic IP to dissoc

Re: [lopsa-tech] Monitoring your AWS instances

2014-10-01 Thread Sean Lally
Interesting, maybe the issue was the zabbix server? If you set it up in a vpc you can hang on to both public and private ip... On Wed, Oct 1, 2014 at 9:57 AM, Edward Ned Harvey (lopser) < lop...@nedharvey.com> wrote: > > From: tech-boun...@lists.lopsa.org [mailto:tech-boun...@lists.lopsa.org] >

Re: [lopsa-tech] Monitoring your AWS instances

2014-10-01 Thread Edward Ned Harvey (lopser)
> From: tech-boun...@lists.lopsa.org [mailto:tech-boun...@lists.lopsa.org] > On Behalf Of Edward Ned Harvey (lopser) > > 2014 Sep: 186 (so far) Grrarrr. Now I'm just bitching (read: venting). Due to alerts being clearly out of control, and presumably related to the xen patching, scheduled to b

Re: [lopsa-tech] Monitoring your AWS instances

2014-09-28 Thread Tom Limoncelli
You might also find this useful: http://blog.scoutapp.com/articles/2013/07/25/understanding-cpu-steal-time-when-should-you-be-worried On Sun, Sep 28, 2014 at 11:42 PM, Charles Polisher wrote: > Tom Throckmorton wrote: > > Charles Polisher wrote: > > > > > Wish I knew a metric to expose CPU cy

Re: [lopsa-tech] Monitoring your AWS instances

2014-09-28 Thread Charles Polisher
Tom Throckmorton wrote: > Charles Polisher wrote: > > > Wish I knew a metric to expose CPU cycles you didn't get because > > some other VM was sucking them up. Maybe L2 cache misses or TLB > > misses would be revealing. Maybe low CPU utilization when you > > should see it getting busy? > > > > CP

Re: [lopsa-tech] Monitoring your AWS instances

2014-09-28 Thread Tom Throckmorton
On Sun, Sep 28, 2014 at 11:42 AM, Charles Polisher wrote: > Wish I knew a metric to expose CPU cycles you didn't get because > some other VM was sucking them up. Maybe L2 cache misses or TLB > misses would be revealing. Maybe low CPU utilization when you > should see it getting busy? > CPU % ste

Re: [lopsa-tech] Monitoring your AWS instances

2014-09-28 Thread Yves Dorfsman
On 2014-09-28 08:54, Edward Ned Harvey (lopser) wrote: > > And "Customers who aren’t sure if they are impacted should go to the “Events“ > page on the EC2 console, which will list any pending instance reboots for > their AWS account." Also you can set "alternate contact" in your "account settin

Re: [lopsa-tech] Monitoring your AWS instances

2014-09-28 Thread Yves Dorfsman
On 2014-09-28 09:00, Edward Ned Harvey (lopser) wrote: >> From: Derek Balling [mailto:dr...@megacity.org] >> Sent: Sunday, September 28, 2014 10:46 AM >> >> Wild-ass Speculation: With Amazon's commitment to cloud computing, and >> with SDN use on the rise... is it possible they're rebooting "virtua

Re: [lopsa-tech] Monitoring your AWS instances

2014-09-28 Thread Charles Polisher
On Sun, Sep 28, 2014 at 03:00:35PM +, Edward Ned Harvey (lopser) wrote: > > From: Derek Balling [mailto:dr...@megacity.org] > > Sent: Sunday, September 28, 2014 10:46 AM > > > > Wild-ass Speculation: With Amazon's commitment to cloud computing, and > > with SDN use on the rise... is it possibl

Re: [lopsa-tech] Monitoring your AWS instances

2014-09-28 Thread Edward Ned Harvey (lopser)
> From: Derek Balling [mailto:dr...@megacity.org] > Sent: Sunday, September 28, 2014 10:46 AM > > Wild-ass Speculation: With Amazon's commitment to cloud computing, and > with SDN use on the rise... is it possible they're rebooting "virtualized > network gear"? Extremely possible. But that ... m

Re: [lopsa-tech] Monitoring your AWS instances

2014-09-28 Thread Edward Ned Harvey (lopser)
> From: Derek Balling [mailto:dr...@megacity.org] > > You're aware that a large number of instances in US-VA-East are getting > rebooted, right? Yes, but none of our systems are scheduled for reboot, *and* the machines triggering our alerts have not been rebooting - they just become unavailable

Re: [lopsa-tech] Monitoring your AWS instances

2014-09-28 Thread Edward Ned Harvey (lopser)
> From: Matthew Barr [mailto:mb...@mbarr.net] > > It's going to be every region over the next few days.  Pretty much every > hypervisor is getting a security patch, including RDS & Elasticcache system > hosts. Amazon says fewer than 10% of systems are affected http://aws.amazon.com/blogs/aws/ec2-

Re: [lopsa-tech] Monitoring your AWS instances

2014-09-28 Thread Derek Balling
On Sep 28, 2014, at 10:44 AM, Edward Ned Harvey (lopser) wrote: > Yes, but none of our systems are scheduled for reboot, *and* the machines > triggering our alerts have not been rebooting - they just become unavailable > on the network for a few minutes and then reappear, without any sort of c

Re: [lopsa-tech] Monitoring your AWS instances

2014-09-28 Thread Matthew Barr
It's going to be every region over the next few days. Pretty much every hypervisor is getting a security patch, including RDS & Elasticcache system hosts. We had over 65 nodes over 4 regions for one small account. Sent from my iPad On Sep 28, 2014, at 9:58 AM, Derek Balling wrote: You're awar

Re: [lopsa-tech] Monitoring your AWS instances

2014-09-28 Thread Derek Balling
You're aware that a large number of instances in US-VA-East are getting rebooted, right? http://siliconangle.com/blog/2014/09/26/amazon-gives-an-update-on-its-impending-ec2-instance-reboot/? On Sep 28, 2014, at 9:55 AM, Edward Ned Harvey (lopser) wrote: > I would really like to hear from an