> Do you find it frustrating that there's no way to turn on instrumentation > of function outputs for manual testing?
Yes. In particular when I'm refactoring code and want to know when I'm returning something surprising. I haven't yet had much success with spec generators for CLJS / UI code. Too many non-clojure types to deal with - my gen-fu is limited. I suspect the argument for reading an instrument flag to turn on return value testing is just that: 1. to help those who aren't able to leverage generators in their problem domain. 2. informal exploration testing I see that spec/assert would give me the same results. That's a fantastic addition but is also a code intrusion. Final though: I suspect you can have it for the cost of a small helper function. The instrument-all code doesn't seem complex but there may be a few private vars between you and happiness. On Sunday, 10 July 2016 20:04:39 UTC+10, puzzler wrote: > > I've played around now with implementing specs for a couple of my projects. > > What I'm finding is that writing specs to check the inputs of a function > is easy-ish, but building useful random generators is very hard -- in my > projects, this seems too hard to do with any reasonable amount of effort. > > This isn't due to anything inherent in clojure.spec, it's just that for > non-trivial functions, coming up with relevant random input is a very hard > problem. For example, let's say I have a function that takes two > integers. Odds are that not any two randomly chosen integers will work. > Some combinations of integers are non-sensical and could trigger an error, > other combinations may cause the function to run way too long. As a > concrete example, I just tried to spec out a SAT solver (which tries to > solve NP-complete problems). The input should be a vector of vectors of > ints, but many combinations of inputs will just run forever. How to > generate "good" SAT problems? I have no idea. > > So for the most part, I've ignored the generation aspect of specs because > it just feels like too much work. But without the generators, > clojure.spec's utility is significantly diminished. > > 1. No way to test function output specs. For documentation purposes, I > want to have an output spec on my function. However, as far as I know, > after instrumentation-triggered checking of output specs was removed a > couple of alphas ago, the only way remaining to check against output specs > is to use randomly generated tests. So if I can't make good generators, I > have no way to confirm that my output spec works the way I think it does. > My documentation could be totally out of touch with reality, and that > displeases me. > > 2. Limited ability for testing that functions take and receive what you > expect. Whereas a static type system can prove certain properties about > whether functions are called with valid inputs, with spec, you only get > those guarantees if you pump a function with enough valid random data to > trigger the function calling all its callees with interesting combinations > of data. But if I use the naive generators, the function will never even > complete with most of the randomly generated input, let alone call other > functions in a useful way. And in many cases I don't see how to generate > something of better quality. > > So looking back at my experiments, my preliminary impression is that by > adding specs to my public APIs, I've gained some useful documentation, and > I've given users the ability to instrument functions in order to get > high-quality assertion-checking of the inputs. In some cases, the error > messages for bad input when instrumented are also more useful than I would > have otherwise gotten, but in some cases they aren't. Overall, I've > struggled to write generators, and without them, the value proposition > isn't as great. > > One other issue I've had, unrelated to generators, is that I'm struggling > to express higher-order type constraints, for example, this function takes > a vector v1 of anything and a vector v2 of anything, but the type of the > things in vector v1 better match the type of the things in vector v2. > > What are other people finding? Do you find it easy/hard to write > generators? (If you think it's easy, I'd love to know your tricks). Do > you find it easy/hard to read specs as a form of documentation about the > contract of a function? Do you find it frustrating that there's no way to > turn on instrumentation of function outputs for manual testing? Do you > feel your generators are providing sufficient code coverage when exercising > callee functions? > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
