Sure; just to clarify, I was definitely talking about capturing test *input*, not output.
BTW, if anyone with a hardware testing background have pointers to good literature on the topic that might be accessible to lowly software twiddlers, fire away... :-) - Chas On Jun 6, 2013, at 6:16 PM, Philip Potter wrote: > I can quite imagine seeing a failure from a Travis ci run which can't be > explained by the test output; being able to reproduce it locally is the first > step towards diagnosis. > > (warning: thread drift ahead...) > > I also have a hardware background - specifically, xilinx FPGAs. The xilinx > synthesis tools have an interesting phenomenon where they use PRNGs in their > place and route algorithms, which map logical netlists to physical slices & > configure the routing to minimize critical path delay. > > A different PRNG seed could result in a different place & route result, with > different maximum clock speed & hence different performance. Hence having a > repeatable build was necessary to ensure uniform performance from the same > source code. > > On Jun 6, 2013 10:38 PM, "Chas Emerick" <c...@cemerick.com> wrote: > Thanks for that perspective, Andy! > > It sounds like one ends up looking for canaries in the coal mine, perhaps > just borderline behaviour or results, odd "sideband emissions", etc. that > indicate that it'd be worthwhile to turn the debug knobs up to 11. It's > interesting to think of cases where the same dynamic might apply with > software where the test data in question isn't readily captured. I can't > think of any right off the top of my head (assuming we're always talking > about purely functional properties under test), but I'm sure they exist. > > Back in the weeds, it does sound like the analogous default would be to vary > the seed for every test run, but always capturing it for replay if necessary. > > - Chas > > On Jun 6, 2013, at 3:52 PM, Andy Fingerhut wrote: > >> I've worked on hardware logic design, where the time and effort required to >> create good tests that find subtle bugs rivals the complexity of the >> hardware being designed itself. In this context, much of the testing has >> often been generated using pseudo-random streams similar to test.generative >> and the simulation approach being advocated for software. I think that is a >> great idea for improving software quality. >> >> In hardware testing via simulation, reproducible tests are important because >> most of the testing cycles are run with a low amount of logging enabled -- >> just enough logging to look for signs of incorrect behavior. When a >> hardware designer wants to debug the problem, the test can be re-run with >> very detailed logs enabled (basically amounting to recording the logic 0/1 >> value of every wire in the hardware being designed at every instant in >> simulated time). These are much slower and require a lot more temporary >> disk space to record the logs. >> >> So you end up using a large variety of different seed values to increase >> test coverage, and if any of them fails, you can turn on the extra logging >> and re-run it, knowing that you will hit the same problem as before. >> >> Andy >> >> >> On Thu, Jun 6, 2013 at 12:30 PM, Chas Emerick <c...@cemerick.com> wrote: >> Well, if *that's* all it is, I'll feel like quite the heel for putting so >> much thought into it! ;-) >> >> Assuming failures are rarer, then starting with a just-previously-failed >> seed would be better as an explicit action, rather than defaulting to a >> constant? >> >> - Chas >> >> On Jun 6, 2013, at 3:21 PM, Raoul Duke wrote: >> >> > i always thought it was basically solely for letting you re-run the >> > test that just/previously failed, nothing more weird or silly than >> > that. >> >> -- >> -- >> You received this message because you are subscribed to the Google >> Groups "Clojure" group. >> To post to this group, send email to clojure@googlegroups.com >> Note that posts from new members are moderated - please be patient with your >> first post. >> To unsubscribe from this group, send email to >> clojure+unsubscr...@googlegroups.com >> For more options, visit this group at >> http://groups.google.com/group/clojure?hl=en >> --- >> You received this message because you are subscribed to the Google Groups >> "Clojure" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to clojure+unsubscr...@googlegroups.com. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> >> >> -- >> -- >> You received this message because you are subscribed to the Google >> Groups "Clojure" group. >> To post to this group, send email to clojure@googlegroups.com >> Note that posts from new members are moderated - please be patient with your >> first post. >> To unsubscribe from this group, send email to >> clojure+unsubscr...@googlegroups.com >> For more options, visit this group at >> http://groups.google.com/group/clojure?hl=en >> --- >> You received this message because you are subscribed to the Google Groups >> "Clojure" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to clojure+unsubscr...@googlegroups.com. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> > > > -- > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with your > first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out. > > > > -- > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with your > first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out. > > -- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
