Re: Data Center testing

2009-08-27 Thread Matthew Palmer
On Wed, Aug 26, 2009 at 03:32:42PM +, Dylan Ebner wrote: > I always love it when I get an outage report from my ISP's or datacenter > and they say an "unexpected issue" or "unforseen issue" caused the > problem. Well, at least it's better than "yeah, we knew about it, but didn't think it was w

Re: Data Center testing

2009-08-26 Thread Warren Kumari
On Aug 24, 2009, at 9:38 AM, Dan Snyder wrote: We have done power tests before and had no problem. I guess I am looking for someone who does testing of the network equipment outside of just power tests. We had an outage due to a configuration mistake that became apparent when a switch f

Re: Data Center testing

2009-08-26 Thread Ross Vandegrift
On Tue, Aug 25, 2009 at 12:53:10PM +, Jeff Aitken wrote: > you have to have some way of describing the desired state of the network in > machine-parsable format Any suggested tools for describing the desired state of the network? NDL, the only option I'm familiar with, is just a brute-force a

Re: Data Center testing

2009-08-26 Thread Jeff Aitken
On Tue, Aug 25, 2009 at 10:45:07PM -0500, Frank Bulk - iName.com wrote: > There's more to data integrity in a data center (well, anything powered, > that is) than network configurations. Understood and agreed. My point was that induced failure testing isn't the right way to catch incorrect or u

RE: Data Center testing

2009-08-26 Thread Deepak Jain
ven a nanosecond when the state-of-the-art says something very different indeed. Deepak Jain AiNET > -Original Message- > From: Dylan Ebner [mailto:dylan.eb...@crlmed.com] > Sent: Wednesday, August 26, 2009 11:33 AM > To: Dan Snyder; Ken Gilmour > Cc: NANOG list > Subject

RE: Data Center testing

2009-08-26 Thread Dylan Ebner
ve it when I get an outage report from my ISP's or datacenter and they say an "unexpected issue" or "unforseen issue" caused the problem. Dylan -Original Message- From: Dan Snyder [mailto:sliple...@gmail.com] Sent: Monday, August 24, 2009 8:39 AM To: Ken Gilmour Cc

Re: Data Center testing

2009-08-26 Thread Jack Bates
James Hess wrote: Config checking can't say much about silent hardware failures. Unanticipated problems are likely to arise in failover systems, especially complicated ones. A failover system that has not been periodically verified may not work as designed. I've seen 3-4 failover failures in

Re: Data Center testing

2009-08-25 Thread eric clark
Most Provider type datacenters I've worked with get a lot of flak from customers when they announce they're doing network failover testing, because there's always going to be a certain amount of chance (at least) of disruption. Its the exception to find a provider that does it I think (or maybe jus

Re: Data Center testing

2009-08-25 Thread James Hess
On Tue, Aug 25, 2009 at 7:53 AM, Jeff Aitken wrote: >[..] Periodically inducing failures to catch [...] them is sorta like using >your smoke detector as an oven timer. >[..] > machine-parsable format, but the benefit is that you know in pseudo-realtime > when something is wrong, as opposed to find

RE: Data Center testing

2009-08-25 Thread Frank Bulk - iName.com
ch. Frank -Original Message- From: Jeff Aitken [mailto:jait...@aitken.com] Sent: Tuesday, August 25, 2009 7:53 AM To: Dan Snyder Cc: NANOG list Subject: Re: Data Center testing On Mon, Aug 24, 2009 at 09:38:38AM -0400, Dan Snyder wrote: > We have done power tests before and had no pr

Re: Data Center testing

2009-08-25 Thread Jeff Aitken
On Mon, Aug 24, 2009 at 09:38:38AM -0400, Dan Snyder wrote: > We have done power tests before and had no problem. I guess I am looking > for someone who does testing of the network equipment outside of just power > tests. We had an outage due to a configuration mistake that became apparent > when

RE: Data Center testing

2009-08-24 Thread Deepak Jain
> At least once a year I like to go out and kick the service entrance > breaker to give the whole enchilada an honest to $diety plugs out test. > As you said, not recommenced if you don't maintain stuff, but that's > how > confident I feel that my system works. Nature has a way of testing it, even

Re: Data Center testing

2009-08-24 Thread Seth Mattinen
Deepak Jain wrote: > Thanks for the kind words Ken. > > Power failure testing and network testing are very different disciplines. > > We operate from the point of view that if a failure occurs because we have > scheduled testing, it is far better since we have the resources on-site to > addres

RE: Data Center testing

2009-08-24 Thread Deepak Jain
Thanks for the kind words Ken. Power failure testing and network testing are very different disciplines. We operate from the point of view that if a failure occurs because we have scheduled testing, it is far better since we have the resources on-site to address it (as opposed to an unplanned

Re: Data Center testing

2009-08-24 Thread Jack Bates
Dan Snyder wrote: We have done power tests before and had no problem. I guess I am looking for someone who does testing of the network equipment outside of just power tests. We had an outage due to a configuration mistake that became apparent when a switch failed. It didn't cause a problem how

Re: Data Center testing

2009-08-24 Thread Dan Snyder
We have done power tests before and had no problem. I guess I am looking for someone who does testing of the network equipment outside of just power tests. We had an outage due to a configuration mistake that became apparent when a switch failed. It didn't cause a problem however when we did a p

Re: Data Center testing

2009-08-24 Thread Ken Gilmour
I know Peer1 in vancouver reguarly send out notifications of "non-impacting" generator load testing, like monthly. Also InterXion in Dublin, Ireland have occasionally sent me notification that there was a power outage of less than a minute however their backup successfully took the load. I only re

RE: [SPAM-HEADER] - Data Center testing - Email has different SMTP TO: and MIME TO: fields in the email addresses

2009-08-24 Thread Rod Beck
"-Original Message- From: Dan Snyder [mailto:sliple...@gmail.com] Sent: Mon 8/24/2009 2:00 PM To: NANOG list Subject: [SPAM-HEADER] - Data Center testing - Email has different SMTP TO: and MIME TO: fields in the email addresses Does any one know of any data centers that do fa

Data Center testing

2009-08-24 Thread Dan Snyder
Does any one know of any data centers that do failure testing of their networking equipment regularly? I mean to verify that everything fails over properly after changes have been made over time. Is there any best practice guides for doing this? Thanks, Dan