I can quite imagine seeing a failure from a Travis ci run which can't be explained by the test output; being able to reproduce it locally is the first step towards diagnosis.
(warning: thread drift ahead...) I also have a hardware background - specifically, xilinx FPGAs. The xilinx synthesis tools have an interesting phenomenon where they use PRNGs in their place and route algorithms, which map logical netlists to physical slices & configure the routing to minimize critical path delay. A different PRNG seed could result in a different place & route result, with different maximum clock speed & hence different performance. Hence having a repeatable build was necessary to ensure uniform performance from the same source code. On Jun 6, 2013 10:38 PM, "Chas Emerick" <c...@cemerick.com> wrote: > Thanks for that perspective, Andy! > > It sounds like one ends up looking for canaries in the coal mine, perhaps > just borderline behaviour or results, odd "sideband emissions", etc. that > indicate that it'd be worthwhile to turn the debug knobs up to 11. It's > interesting to think of cases where the same dynamic might apply with > software where the test data in question isn't readily captured. I can't > think of any right off the top of my head (assuming we're always talking > about purely functional properties under test), but I'm sure they exist. > > Back in the weeds, it does sound like the analogous default would be to > vary the seed for every test run, but always capturing it for replay if > necessary. > > - Chas > > On Jun 6, 2013, at 3:52 PM, Andy Fingerhut wrote: > > I've worked on hardware logic design, where the time and effort required > to create good tests that find subtle bugs rivals the complexity of the > hardware being designed itself. In this context, much of the testing has > often been generated using pseudo-random streams similar to test.generative > and the simulation approach being advocated for software. I think that is > a great idea for improving software quality. > > In hardware testing via simulation, reproducible tests are important > because most of the testing cycles are run with a low amount of logging > enabled -- just enough logging to look for signs of incorrect behavior. > When a hardware designer wants to debug the problem, the test can be re-run > with very detailed logs enabled (basically amounting to recording the logic > 0/1 value of every wire in the hardware being designed at every instant in > simulated time). These are much slower and require a lot more temporary > disk space to record the logs. > > So you end up using a large variety of different seed values to increase > test coverage, and if any of them fails, you can turn on the extra logging > and re-run it, knowing that you will hit the same problem as before. > > Andy > > > On Thu, Jun 6, 2013 at 12:30 PM, Chas Emerick <c...@cemerick.com> wrote: > >> Well, if *that's* all it is, I'll feel like quite the heel for putting so >> much thought into it! ;-) >> >> Assuming failures are rarer, then starting with a just-previously-failed >> seed would be better as an explicit action, rather than defaulting to a >> constant? >> >> - Chas >> >> On Jun 6, 2013, at 3:21 PM, Raoul Duke wrote: >> >> > i always thought it was basically solely for letting you re-run the >> > test that just/previously failed, nothing more weird or silly than >> > that. >> >> -- >> -- >> You received this message because you are subscribed to the Google >> Groups "Clojure" group. >> To post to this group, send email to clojure@googlegroups.com >> Note that posts from new members are moderated - please be patient with >> your first post. >> To unsubscribe from this group, send email to >> clojure+unsubscr...@googlegroups.com >> For more options, visit this group at >> http://groups.google.com/group/clojure?hl=en >> --- >> You received this message because you are subscribed to the Google Groups >> "Clojure" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to clojure+unsubscr...@googlegroups.com. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> > > -- > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out. > > > > > -- > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out. > > > -- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
