In case anyone was wondering, it looks like this is probably a known issue:
https://dev.clojure.org/jira/browse/CLJ-2079 On Wednesday, February 28, 2018 at 12:05:31 AM UTC-6, James Gatannah wrote: > > > > On Tuesday, February 27, 2018 at 12:45:26 AM UTC-6, Didier wrote: >> >> Don't fully understand what you are doing, >> > > (Slightly) bigger picture, I have a pair of functions that > serialize/deserialize maps where the values are mostly fixed-length byte > arrays. > > I want to verify that they round-trip correctly. > > This specific example is about two of those fields which really have the > same spec. I'm trying to define this in a way that's similar to a C typedef > (or possibly just something like using OOP inheritance for the sake of > documentation): > > (s/def ::extension ...) > (s/def ::server-extension ::extension) > (s/def ::client-extension ::extension) > > I'm doing this in an attempt to trying to reduce my nesting levels. > > This is what the weird subject is about: I'm trying to take advantage of > the transitive property of specs. > > > >> but when you run test.check, it seems to not always be generating random >> sample, it'll grow the samples. When you run sample, it'll always start >> back from the beginning. >> > >> You can see that by running: >> >> (sgen/sample (s/gen ::test) 100) >> >> See how the generated samples are getting longer and longer? >> > > Now instead try running this 10 times: > (sgen/sample (s/gen ::test) 10) > > And notice how you always get the same short samples? > > > > (I'm sorry about the quote mangling). > > Since I have to have a specific length, I'm specifying that as a parameter > to sgen/vector. > > This part of it works. I can run s/sample on my generator and validate its > output against the spec without any trouble. > > Actually, if I define the ::extension spec using with-gen, it also looks > as though it works (I just ran across this, so haven't tested it > thoroughly). But then my library has a runtime dependency on test.check. > That's not the end of the world, but it's a definite downside. > > I haven't been able to find real documentation covering the arguments to > s/gen. But it looks as though I can pass in a map of 0-arity functions to > override the different generators. The keys to that map override specs > being generated. I can't remember now where I found this, but it can't be > totally undocumented. I haven't studied the source code thoroughly enough > to have picked out that sort of detail. > > So: > > (s/gen (s/keys :req [::server-extension > ::client-extension]) > {::server-extension #(gen/fmap byte-array (gen/vector (gen/choose > -128 127) extension-length)) > ::client-extension #(gen/fmap byte-array (gen/vector (gen/choose > -128 127) extension-length))}) > > > calls the function in the value associated with each key to override the > generator for that spec. > > This is the part that seems to only work sometimes. > > When I extend this to include all the keys in the full spec, these are the > only two that have problems. They're also the only ones that are directly > defined as another spec definition. (Which is why I'm pointing my finger at > my attempt to use the "transitive" property, though that could totally be a > red herring). > > When I run this inside deftest, it seems fine. Really, that's all I need. > But I'd really like to understand the seemingly non-deterministic failures. > > I wrote a function called manual-check that takes that generator and calls > sgen/sample on it. > > When I call that function directly from the REPL, it very nearly always > fails. > > When I call it 40 times in a reduce inside deftest, I haven't seen it fail > yet. > > * While I was writing this, I stumbled across another piece that seems > like an interesting clue: > > If I override the generator for ::extension rather than the individual > specs that I've defined transitively, it seems to start working. > > i.e. > > (s/gen (s/keys :req [::server-extension > ::client-extension]) > {::extension #(gen/fmap byte-array (gen/vector (gen/choose -128 > 127) extension-length))}) > > seems to work. > > It's still less than ideal, in case I ever want to change the underlying > spec that's the "base" definition, since I'll have to remember to change > the base generator. But I really shouldn't have this sort of thing > scattered around everywhere. Especially since it's most interesting inside > deftest anyway, where both approaches always seem to work (I haven't tested > this out thoroughly yet either). > > My current hypothesis is that there's something about the deftest macro > that interacts with test.check in a way that makes gen more capable in > terms of resolving which spec I actually mean/want to override. I know > (based on digging through internals) that it calls something (I think it's > named specize) that resolves the spec's name. Then it has to use that > name/those names in the overriding map parameter to pick out the > appropriate function to call to get the generator. > > I'm very skeptical, but could that possibly hold water? > > Thank you to anyone who actually took the time to read this! > > > >> >> >> On Monday, 26 February 2018 21:39:09 UTC-8, James Gatannah wrote: >>> >>> Fairly minimalist example available at >>> https://gist.github.com/jimrthy/21851c52a8cd6b04a31ed08b1d0a7f04 >>> >>> When I call gen/sample from inside a unit test, it seems to pass with >>> flying colors. >>> >>> When I directly eval the gen/sample form or call (manual-check) from the >>> REPL (I checked both CIDER and the boot CLI, just in case), I usually get >>> the "Couldn't satisfy such-that predicate after 100 tries." >>> >>> To be a little more specific about this: >>> >>> Calling (manual-check) failed 49/50 times. >>> >>> Calling (transitive-indirect) passed 50 times in a row. If you haven't >>> bothered looking at the gist, each of those calls (manual-check) 40 times. >>> >>> I was quite surprised by this. Does anyone have any suggestions about >>> why wrapping the call in a unit test might help it succeed more often? >>> >>> If I just call gen/sample on the generator I'm trying to use, followed >>> by s/valid? for one of the specs I'm trying to generate, it seems fine. I >>> haven't dug into the source here (and I'm not positive what all's going on >>> inside the spec.gen namespace), but I thought that's what gen/generate does >>> when you define a custom generator for your spec. >>> >>> Except that isn't really what I'm doing. >>> >>> I'm trying to avoid adding extra runtime dependencies on a library like >>> tools.check, so I'm trying to do this with overrides in the test namespaces >>> to try to limit the extra dependencies to test time. >>> >>> Could that be where I'm breaking core assumptions that don't seem to >>> cause trouble for anyone else? >>> >>> Thanks, >>> James >>> >>> >>> >>> On Sunday, February 25, 2018 at 6:45:39 PM UTC-6, James Gatannah wrote: >>>> >>>> I have a spec for an array of 16 bytes: >>>> >>>> (s/def ::extension (s/and bytes? >>>> #(= (count %) 16)) >>>> >>>> Then I have a couple of other specs that are really just renaming it: >>>> >>>> (s/def ::client-extension ::extension) >>>> (s/def ::server-extension ::extension) >>>> >>>> I started doing some refactoring today, and the definitions wound up >>>> needing to move to a different namespace. >>>> >>>> So now the original definitions have changed to >>>> >>>> (s/def ::client-extension ::refactored/client-extension) >>>> >>>> I also started dabbling with generators, and came up with this: >>>> >>>> (gen/generate (s/gen ::client-extension >>>> {::client-extension #(gen/fmap >>>> byte-array (gen/vector (gen/choose -128 127) 16)})) >>>> >>>> When I define things this way, I get a "Couldn't satisfy such-that >>>> predicate after 100 tries." exception a little more than half the time. >>>> >>>> If I rearrange things so that either >>>> a) The refactored namespace defines the spec directly >>>> or >>>> b) I change my generator override to specify the top-level spec that >>>> the others are copying >>>> >>>> i.e. >>>> a) would mean changing the refactored ns such that I have >>>> (s/def ::client-extension (s/and bytes? >>>> #(= (count %) 16)) >>>> >>>> b) changing the generator to >>>> (gen/generate (s/gen ::client-extension >>>> {::refactored/extension #(gen/fmap >>>> byte-array (gen/vector (gen/choose -128 127) 16)})) >>>> >>>> it seems to fail (with the same problem) about 1 time in 5. >>>> >>>> I haven't seen it fail yet if I undo my refactoring and move the spec >>>> back to the original location. >>>> >>>> I haven't collected any sorts of real numbers on this, much less tried >>>> to make enough test runs to collect a statistically significant sample. I >>>> know the next real steps are to put together a minimalist example. >>>> >>>> But before I do that, I figured it might be asking whether anyone sees >>>> anything obviously wrong in what I'm trying to do, or whether there's a >>>> better way to do it. >>>> >>>> Thanks in advance, >>>> James >>>> >>>> > On Tuesday, February 27, 2018 at 12:45:26 AM UTC-6, Didier wrote: >> >> Don't fully understand what you are doing, but when you run test.check, >> it seems to not always be generating random sample, it'll grow the samples. >> When you run sample, it'll always start back from the beginning. >> >> You can see that by running: >> >> (sgen/sample (s/gen ::test) 100) >> >> See how the generated samples are getting longer and longer? >> >> Now instead try running this 10 times: >> >> (sgen/sample (s/gen ::test) 10) >> >> And notice how you always get the same short samples? >> >> >> On Monday, 26 February 2018 21:39:09 UTC-8, James Gatannah wrote: >>> >>> Fairly minimalist example available at >>> https://gist.github.com/jimrthy/21851c52a8cd6b04a31ed08b1d0a7f04 >>> >>> When I call gen/sample from inside a unit test, it seems to pass with >>> flying colors. >>> >>> When I directly eval the gen/sample form or call (manual-check) from the >>> REPL (I checked both CIDER and the boot CLI, just in case), I usually get >>> the "Couldn't satisfy such-that predicate after 100 tries." >>> >>> To be a little more specific about this: >>> >>> Calling (manual-check) failed 49/50 times. >>> >>> Calling (transitive-indirect) passed 50 times in a row. If you haven't >>> bothered looking at the gist, each of those calls (manual-check) 40 times. >>> >>> I was quite surprised by this. Does anyone have any suggestions about >>> why wrapping the call in a unit test might help it succeed more often? >>> >>> If I just call gen/sample on the generator I'm trying to use, followed >>> by s/valid? for one of the specs I'm trying to generate, it seems fine. I >>> haven't dug into the source here (and I'm not positive what all's going on >>> inside the spec.gen namespace), but I thought that's what gen/generate does >>> when you define a custom generator for your spec. >>> >>> Except that isn't really what I'm doing. >>> >>> I'm trying to avoid adding extra runtime dependencies on a library like >>> tools.check, so I'm trying to do this with overrides in the test namespaces >>> to try to limit the extra dependencies to test time. >>> >>> Could that be where I'm breaking core assumptions that don't seem to >>> cause trouble for anyone else? >>> >>> Thanks, >>> James >>> >>> >>> >>> On Sunday, February 25, 2018 at 6:45:39 PM UTC-6, James Gatannah wrote: >>>> >>>> I have a spec for an array of 16 bytes: >>>> >>>> (s/def ::extension (s/and bytes? >>>> #(= (count %) 16)) >>>> >>>> Then I have a couple of other specs that are really just renaming it: >>>> >>>> (s/def ::client-extension ::extension) >>>> (s/def ::server-extension ::extension) >>>> >>>> I started doing some refactoring today, and the definitions wound up >>>> needing to move to a different namespace. >>>> >>>> So now the original definitions have changed to >>>> >>>> (s/def ::client-extension ::refactored/client-extension) >>>> >>>> I also started dabbling with generators, and came up with this: >>>> >>>> (gen/generate (s/gen ::client-extension >>>> {::client-extension #(gen/fmap >>>> byte-array (gen/vector (gen/choose -128 127) 16)})) >>>> >>>> When I define things this way, I get a "Couldn't satisfy such-that >>>> predicate after 100 tries." exception a little more than half the time. >>>> >>>> If I rearrange things so that either >>>> a) The refactored namespace defines the spec directly >>>> or >>>> b) I change my generator override to specify the top-level spec that >>>> the others are copying >>>> >>>> i.e. >>>> a) would mean changing the refactored ns such that I have >>>> (s/def ::client-extension (s/and bytes? >>>> #(= (count %) 16)) >>>> >>>> b) changing the generator to >>>> (gen/generate (s/gen ::client-extension >>>> {::refactored/extension #(gen/fmap >>>> byte-array (gen/vector (gen/choose -128 127) 16)})) >>>> >>>> it seems to fail (with the same problem) about 1 time in 5. >>>> >>>> I haven't seen it fail yet if I undo my refactoring and move the spec >>>> back to the original location. >>>> >>>> I haven't collected any sorts of real numbers on this, much less tried >>>> to make enough test runs to collect a statistically significant sample. I >>>> know the next real steps are to put together a minimalist example. >>>> >>>> But before I do that, I figured it might be asking whether anyone sees >>>> anything obviously wrong in what I'm trying to do, or whether there's a >>>> better way to do it. >>>> >>>> Thanks in advance, >>>> James >>>> >>>> -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
