Do you find it frustrating that there's no way to turn on instrumentation of function outputs for manual testing?
Yes. On Mon, Jul 11, 2016 at 4:16 PM, Oliver George <oli...@condense.com.au> wrote: > > Do you find it frustrating that there's no way to turn on instrumentation >> of function outputs for manual testing? > > > Yes. > > In particular when I'm refactoring code and want to know when I'm > returning something surprising. > > I haven't yet had much success with spec generators for CLJS / UI code. > Too many non-clojure types to deal with - my gen-fu is limited. > > I suspect the argument for reading an instrument flag to turn on return > value testing is just that: > > 1. to help those who aren't able to leverage generators in their > problem domain. > 2. informal exploration testing > > I see that spec/assert would give me the same results. That's a fantastic > addition but is also a code intrusion. > > Final though: I suspect you can have it for the cost of a small helper > function. The instrument-all code doesn't seem complex but there may be a > few private vars between you and happiness. > > > On Sunday, 10 July 2016 20:04:39 UTC+10, puzzler wrote: >> >> I've played around now with implementing specs for a couple of my >> projects. >> >> What I'm finding is that writing specs to check the inputs of a function >> is easy-ish, but building useful random generators is very hard -- in my >> projects, this seems too hard to do with any reasonable amount of effort. >> >> This isn't due to anything inherent in clojure.spec, it's just that for >> non-trivial functions, coming up with relevant random input is a very hard >> problem. For example, let's say I have a function that takes two >> integers. Odds are that not any two randomly chosen integers will work. >> Some combinations of integers are non-sensical and could trigger an error, >> other combinations may cause the function to run way too long. As a >> concrete example, I just tried to spec out a SAT solver (which tries to >> solve NP-complete problems). The input should be a vector of vectors of >> ints, but many combinations of inputs will just run forever. How to >> generate "good" SAT problems? I have no idea. >> >> So for the most part, I've ignored the generation aspect of specs because >> it just feels like too much work. But without the generators, >> clojure.spec's utility is significantly diminished. >> >> 1. No way to test function output specs. For documentation purposes, I >> want to have an output spec on my function. However, as far as I know, >> after instrumentation-triggered checking of output specs was removed a >> couple of alphas ago, the only way remaining to check against output specs >> is to use randomly generated tests. So if I can't make good generators, I >> have no way to confirm that my output spec works the way I think it does. >> My documentation could be totally out of touch with reality, and that >> displeases me. >> >> 2. Limited ability for testing that functions take and receive what you >> expect. Whereas a static type system can prove certain properties about >> whether functions are called with valid inputs, with spec, you only get >> those guarantees if you pump a function with enough valid random data to >> trigger the function calling all its callees with interesting combinations >> of data. But if I use the naive generators, the function will never even >> complete with most of the randomly generated input, let alone call other >> functions in a useful way. And in many cases I don't see how to generate >> something of better quality. >> >> So looking back at my experiments, my preliminary impression is that by >> adding specs to my public APIs, I've gained some useful documentation, and >> I've given users the ability to instrument functions in order to get >> high-quality assertion-checking of the inputs. In some cases, the error >> messages for bad input when instrumented are also more useful than I would >> have otherwise gotten, but in some cases they aren't. Overall, I've >> struggled to write generators, and without them, the value proposition >> isn't as great. >> >> One other issue I've had, unrelated to generators, is that I'm struggling >> to express higher-order type constraints, for example, this function takes >> a vector v1 of anything and a vector v2 of anything, but the type of the >> things in vector v1 better match the type of the things in vector v2. >> >> What are other people finding? Do you find it easy/hard to write >> generators? (If you think it's easy, I'd love to know your tricks). Do >> you find it easy/hard to read specs as a form of documentation about the >> contract of a function? Do you find it frustrating that there's no way to >> turn on instrumentation of function outputs for manual testing? Do you >> feel your generators are providing sufficient code coverage when exercising >> callee functions? >> > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
