It might be possible to leverage something like American Fuzzy Lop[1] for better random input generation. I've never used AFL myself, but I know SQLite (one of the best-tested libraries I know of) has had good success with it[2], and it does work on Java.
[1] http://lcamtuf.coredump.cx/afl/ [2] https://www.sqlite.org/testing.html#aflfuzz > On Jul 10, 2016, at 03:04, Mark Engelberg <mark.engelb...@gmail.com> wrote: > > I've played around now with implementing specs for a couple of my projects. > > What I'm finding is that writing specs to check the inputs of a function is > easy-ish, but building useful random generators is very hard -- in my > projects, this seems too hard to do with any reasonable amount of effort. > > This isn't due to anything inherent in clojure.spec, it's just that for > non-trivial functions, coming up with relevant random input is a very hard > problem. For example, let's say I have a function that takes two integers. > Odds are that not any two randomly chosen integers will work. Some > combinations of integers are non-sensical and could trigger an error, other > combinations may cause the function to run way too long. As a concrete > example, I just tried to spec out a SAT solver (which tries to solve > NP-complete problems). The input should be a vector of vectors of ints, but > many combinations of inputs will just run forever. How to generate "good" > SAT problems? I have no idea. > > So for the most part, I've ignored the generation aspect of specs because it > just feels like too much work. But without the generators, clojure.spec's > utility is significantly diminished. > > 1. No way to test function output specs. For documentation purposes, I want > to have an output spec on my function. However, as far as I know, after > instrumentation-triggered checking of output specs was removed a couple of > alphas ago, the only way remaining to check against output specs is to use > randomly generated tests. So if I can't make good generators, I have no way > to confirm that my output spec works the way I think it does. My > documentation could be totally out of touch with reality, and that displeases > me. > > 2. Limited ability for testing that functions take and receive what you > expect. Whereas a static type system can prove certain properties about > whether functions are called with valid inputs, with spec, you only get those > guarantees if you pump a function with enough valid random data to trigger > the function calling all its callees with interesting combinations of data. > But if I use the naive generators, the function will never even complete with > most of the randomly generated input, let alone call other functions in a > useful way. And in many cases I don't see how to generate something of > better quality. > > So looking back at my experiments, my preliminary impression is that by > adding specs to my public APIs, I've gained some useful documentation, and > I've given users the ability to instrument functions in order to get > high-quality assertion-checking of the inputs. In some cases, the error > messages for bad input when instrumented are also more useful than I would > have otherwise gotten, but in some cases they aren't. Overall, I've > struggled to write generators, and without them, the value proposition isn't > as great. > > One other issue I've had, unrelated to generators, is that I'm struggling to > express higher-order type constraints, for example, this function takes a > vector v1 of anything and a vector v2 of anything, but the type of the things > in vector v1 better match the type of the things in vector v2. > > What are other people finding? Do you find it easy/hard to write generators? > (If you think it's easy, I'd love to know your tricks). Do you find it > easy/hard to read specs as a form of documentation about the contract of a > function? Do you find it frustrating that there's no way to turn on > instrumentation of function outputs for manual testing? Do you feel your > generators are providing sufficient code coverage when exercising callee > functions? > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with your > first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
