Saku, I think for us - the testing we do is to validate against our own configurations and our own designs. This is not done hand in hand with the vendor - it is done against what we use every day as a starting point - to give us some indication of where the trip points potentially are.
And you are 100% correct - the vendors spend huge amounts testing - but it is impossible for any vendor to test every scenario of every operator out there - which is why I advocate for operators to test against their specific needs - those tests can - and do - show up things occasionally that can be rectified. That testing as I said - we do in our own labs - against our own templates - and our own methodologies - and its worked pretty well for us so far. Andrew -----Original Message----- From: Saku Ytti <[email protected]> Sent: Wednesday, 2 September 2020 10:37 To: Andrew Alston <[email protected]> Cc: [email protected]; Kody Vicknair <[email protected]>; Roger Wiklund <[email protected]>; Colton Conor <[email protected]>; Juniper List <[email protected]> Subject: Re: [j-nsp] How to pick JUNOS Version On Wed, 2 Sep 2020 at 10:23, Andrew Alston <[email protected]> wrote: > 2. Start looking at the new features - decide what may be useful - > if anything - and start testing to that to death - again preferably > before release so that the fixes can be in when it is released How do people measure this? Vendors spend tens or hundreds millions annually on testing, and still deliver absolute trash NOS, to every vendor, and there is no change that I can observe +20 years in quality. Basic things are broken, and everyone finds new fundamental bugs all the time. I think NOS are shit, because shit NOS is a good business case and good NOS is a bad business case, I know it sounds outrageous, but let me explain. Vendor revenue is support contract, not HW sales. And a lot of us don't need help on configuring or troubleshooting, a lot of us have access to community which outperforms TAC on how to get that box working. But none of us has access to the code, we can't commit and push a fix. If the NOS would work, like Windows, Macos or Linux that you rarely find bugs, a lot of us would opt out from support contracts, and would just buy spare HW, destroying the vendor's business case. I don't think vendors sit in scary skull towers and plans for shit NOS, I think it's emergent behaviour from how the market is modelled. And there are ways I think the market could change, but I'm already venturing too far from the question to explore that topic. Now when it comes to testing, many claim it is important and it matters. I'm not convinced. And I don't think people are looking at this in any formality, it's more like religion, and its utility is to improve comfort-to-deploy in the organisation, it doesn't do much towards probability-of- success in my mind. I've worked for companies who test not at all, companies who boot it in the lab and companies who had a team doing just testing, and I can't say I've seen different amounts of TAC cases on software issues. People who invest lot on testing, and are not comfortable with idea that value is just 'comfort-to-deploy' (that may be sufficiently important value), I recommend looking at TAC cases you had which actually did cause customer outage, then try to evaluate 'was this reasonable to catch in the lab', try to be honest. The problem I see, whole NOS quality is shit, it's not so shit that it's always broken, the problems that manifest require usually more than one condition, then if you start to do back-of-the-envelope math on testing everything with every permutations, you will notice no amount of money can fix the fact that you're limited by heat-death-of-universe on the wall clock. So now you're venturing into an area where you gotta choose, what to test and what not to test, and you don't have nearly enough outages and faults to apply statistical analysis on it, so you're actually just guessing. It's religion, which has some utility, but not the utility we think it has. Note I'm not saying testing wholesale is useless, I'm more saying it has an exponentially or worse diminishing return. I would say push 1 packet through all your products in the lab, and you're done, you're as far as you're reasonably gonna get. And start thinking in terms 'the NOS is shit and I exercise no power over it', what actions work in that world? Staging pop with real but outage insensitive subscriptions? -- ++ytti _______________________________________________ juniper-nsp mailing list [email protected] https://puck.nether.net/mailman/listinfo/juniper-nsp
