On 2016-03-15 01:46 -0700, Hal Murray wrote: > I'm not sure what you have in mind for "test results". How is a non-wizard > going to be able to evaluate anything other than a yes/no, and we will > already have filtered out the no-s.
That is all they need to know really. The purpose of the test and whether it passed. It can be as simple as the test page for a build showing all green. For now the existence of a snapshot for that revision on the FTP will denote it passing all the tests. > What is the purpose of a release? > > I'm assuming it's a tag on a pile of bits so we have a way to talk about them > and a way to focus testing and usage on a smaller number of things to keep > track of and support. That and comfort. Users rely on us to say 'we believe this is a good set of changes'. A revision passing all the tests doesn't make it safe. It could be in the middle of a WIP that created a bug. Even when you try not to break up work this way it can happen. A release is a safe checkpoint between work. > The idea of "recommended" sounds good. It looks like you are automating the > 2 week-ish release cycle. We still have to work out which releases to > support long term and how long. Automating the testing, yes. Typically with these types of systems you choose a release that is the most 'stable'. Eg, the last X commits have passed all tests. We can decide that that number will be but at a minimum right now it looks like a full test cycle will take a week. Which means all commits within that week will land in the next (live) test run. I can shrink this down if I purchase more RPIs that is the only limiting factor -- real hardware to run ntpd on. > In terms of testing... > > I look at a lot of graphs by hand. I think I could automate some of that, > but that would still leave a lot of work. Can you send me an email detailing what you do and a line-by-line list of commands you run and what you do with the output? It shouldn't be difficult to automate it. > It may not be possible (or worth the effort) to totally automate the testing. We can see.. if it's easy there is no harm in adding it. > This will probably change when Eric's TESTFRAME starts working. (I'm > assuming we can run some real servers setup to collect data so we can gather > a bunch of test cases when "interesting" things happen.) > > We also have to test refclocks. Yes, I am in contact with a local company they have some RF shielded boxes and GPS signal generators. This would let me put a refclock in one of these boxes and generate a custom GPS signal to pickup and test against. I can pickup some of the cheaper refclocks to test and it would also let us exercise all the common code. > We can get real traffic by putting up pool servers. Yes.. I want to do this I have a really good and what I think is secure design down for handling the pool. I also have a system to detect fake instances that harvest IPs for scanning. I have the groundwork done for this already. Amar. _______________________________________________ devel mailing list devel@ntpsec.org http://lists.ntpsec.org/mailman/listinfo/devel
