[ANN] Citrus (Scrum) 3.0.0 — State management library for Rum

[Roman Liutikov]

Just renaming

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Clojure In the UK: Yapster

[Jon Pither]

Hi All,

Have a read about Clojure being used to build a corporate chat platform and
the tech used: https://juxt.pro/blog/posts/clojure-in-yapster.html
,

Regards,

Jon.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Sum types in Clojure? Better to represent as tagged records or as variant vectors?

[Tim Gilbert]

I've used tagged unions to represent lookup identifiers for variant types 
and found them to be pleasant to work with. I think part of the reason they 
don't seem useful in the context of this discussion is that many of the 
examples given have not actually been variants. For example, in [:image/file 
"/images/image" :jpeg], every value will have those three data elements, 
and for {:status :response :result val}, every value will have both of 
those fields, and their values will be the same type. The identifiers I was 
working with were in heterogenous collections, so I might have [:image/ref 
45] or [:user/name "bob"] or [:project/element "project-slug" 435].

It would certainly be just as easy to use {:collection/type ::person 
:user/name "bob"} and {:collection/type ::project :project/slug 
"project-slug" :project/element 435}, but I like the concision of the 
tagged unions.

On Friday, August 25, 2017 at 4:30:37 PM UTC-4, tbc++ wrote:
>
> >> they're a little nicer to type and read
>
> And that's where I have to disagree. The problem with most of these 
> options is they complect ordering with meaning. 
>
> [:image/file "/images/image" :jpeg]
>
> Here I have no idea what these values mean. I have to have out-of-band 
> information about what offset in the vector corresponds to what value. 
> Functions have this same problem, look no further than `cons` vs `conj` to 
> see potential confusion on argument ordering. 
>
> So why don't we only use maps for function arguments? Well mostly to make 
> the functions easier to manipulate by humans. But some of the best 
> libraries for web APIs (AWS) that I've used have used almost an exclusively 
> map based interface. 
>
> Once again I have to repeat my mantra about DSLs. Don't start with DSLs. 
> Start with maps. Build everything off of maps. They're extensible, easy to 
> introspect, and can drive a host of metaprogramming algorithms. If maps are 
> to hard to understand, build constructor functions on top of them. Then 
> finally build a DSL on top, if you need it. 
>
> Frankly, I have so many things I have to remember during programming. I'd 
> much rather see a very uniform map-based interface. Than any sort of nested 
> vectors, tagged values, or anything else. 
>
> Surely we can't say that this:
> >> [[:image/file :image/web] :image.file/path "/images/image" :
> image.file/format :jpeg :image.web/url "www.myimage.com/image"]
>
> Is a better interface than:
>
> {:image.file/path "/images/image"
>  :image.file/format :jpeg
>  :image.web/url "www.myimage.com/image"}
>
> And as I said before, spec is designed from the start to support data in 
> this format. Stating "this is a file", "this is a web image". Is just book 
> keeping that doesn't need to be done. Is a map a :image/web? Well check its 
> members and see if they match the spec. If they match, then you have a 
> :image/web. No need for sum types, tags, wrapping values in vectors. Simply 
> state what a thing should have for it to conform to an interface, and 
> forget whatever else might be in there. It couldn't be simpler. 
>
> On Fri, Aug 25, 2017 at 11:59 AM, Didier > 
> wrote:
>
>> Thanks for everyone's great input.
>>
>> Currently, I see the big distinction being concise vs extension. Maybe 
>> for streaming variants would be better as the format would be smaller to 
>> transfer over a wire. And for small short lived programs, or things you 
>> know won't need extension, variants offer a slightly more convenient 
>> structure to work with.
>>
>> I think both can be specced easily. S/or a few s/tuple to spec a variant. 
>> And multi-spec over a set for the map version.
>>
>> I'd like to explore then the issue of extensibility with variants. How 
>> would you extend them, is there really no way? This is some of my 
>> brainstorming thoughts.
>>
>> 1) How to design a variant of more then one value?
>>
>>
>> 1.1)
>>
>> I know this is a standard variant of one value:
>>
>> [:image/web "www.myimage.com/image.jpg"]
>>
>> I believe to extend it to more values, you would do:
>>
>> [:image/file "/images/image" :jpeg]
>>
>> That is, you'd threat it as a constructor function, which creates an 
>> :image/file type and takes an ordered list of arguments. This way, each 
>> variant type can even be overloaded on their arguments the way you'd 
>> overload a function.
>>
>> [:image/file "/images/image"]
>>
>> Can default to format :png for example, when using the one arg 
>> constructor.
>>
>> 1.2)
>>
>> An alternative is to keep variants as vector pairs, and nest them.
>>
>> [:image/file [:image/file-path "/images/image"] [:image/file-format:jpeg]]
>>
>> In this form, variants are more like their map counterpart. Each element 
>> is named and itself a variant.
>>
>> 1.3)
>>
>> You could also decide to limit variants to strict pairs, so the second 
>> element of any variant is either a variant or a vector of variants.
>>
>> [:image/file [[:image/file-path "/images/image"] 
>> [:image/file-format:jpeg]]]
>>
>>

Re: [ANN] Clojure 1.9.0-alpha18

[Bozhidar Batsov]

Just for the sake of completeness with regards to the issue in CIDER - it
has been addressed in master and I'll issue a bugfix release soon.

On 24 August 2017 at 05:20, Alex Miller  wrote:

> Correct, this was just something Rich ran into while doing the pluggable
> resolver work. The intent has always been that only aliases were to be
> supported in auto-resolved keyword qualifiers and fully-qualified keywords
> there would accidentally work. The spec and code now *only* support
> aliases.
>
>
>
> On Wed, Aug 23, 2017 at 9:01 PM, Sean Corfield  wrote:
>
>> The breakage in CIDER is a good example of what this change disallows:
>>
>>
>>
>> java.lang.RuntimeException: Invalid token: ::clojure.test/once-fixtures
>>
>>
>>
>> Invalid because clojure.test is not an alias – so it should be
>> :clojure.test/once-fixtures instead (or ::test/once-fixtures). See the
>> source here:
>>
>>
>>
>> https://github.com/clojure-emacs/cider-nrepl/blob/master/src
>> /cider/nrepl/middleware/test.clj#L124-L138
>>
>>
>>
>> Sean Corfield -- (970) FOR-SEAN -- (904) 302-SEAN
>> An Architect's View -- http://corfield.org/
>>
>> "If you're not annoying somebody, you're not really alive."
>> -- Margaret Atwood
>>
>>
>>
>> *From: *Colin Fleming 
>> *Sent: *Wednesday, August 23, 2017 5:59 PM
>> *To: *clojure@googlegroups.com
>> *Subject: *Re: [ANN] Clojure 1.9.0-alpha18
>>
>>
>>
>> Tighten autoresolved keywords and autoresolved namespace map syntax to
>> support *only* aliases, as originally intended
>>
>>
>>
>> What does this mean? Is there a JIRA discussion about this?
>>
>>
>>
>> On 24 August 2017 at 04:03, Alex Miller  wrote:
>>
>> Clojure 1.9.0-alpha18 is now available.
>>
>>
>>
>> Try it via
>>
>>
>>
>> - Download: https://repo1.maven.org/maven2/org/clojure/clojure
>> /1.9.0-alpha18
>>
>> - Leiningen: [org.clojure/clojure "1.9.0-alpha18"]
>>
>>
>>
>> 1.9.0-alpha18 includes the following changes since 1.9.0-alpha17:
>>
>>
>>
>> - Can now bind *reader-resolver* to an impl of LispReader$Resolver to
>> control the reader's use of namespace interactions when resolving
>> autoresolved keywords and maps.
>>
>> - Tighten autoresolved keywords and autoresolved namespace map syntax to
>> support *only* aliases, as originally intended
>>
>> - Updated to use core.specs.alpha 0.1.24
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clojure@googlegroups.com
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+unsubscr...@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to clojure+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clojure@googlegroups.com
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+unsubscr...@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to clojure+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clojure@googlegroups.com
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+unsubscr...@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> ---
>> You received this message because you are subscribed to a topic in the
>> Google Groups "Clojure" group.
>> To unsubscribe from this topic, visit https://groups.google.com/d/to
>> pic/clojure/rb22V98rPLM/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> clojure+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this

core.async got in a bad state?

[Aaron Iba]

My company has a production system that uses core.async extensively. We've 
been running it 24/7 for over a year with occasional restarts to update 
things and add features, and so far core.async has been working great.

The other day, during a particularly high workload, the whole system got 
locked up. All the channels seemed blocked at once.  I was able to connect 
with a REPL and poke around, and noticed strange behavior of core.async. 
Specifically, the following code, when evaluated in the REPL, blocked on 
the put (third expression):

(def c (async/chan))
(go-loop []
  (when-some [x (!! c true)

Whereas on any fresh system, the above expressions obviously succeed.

Puts succeeded if they went onto the channel's buffer, but not when they 
should go through to a consumer. For example with the following 
expressions, evaluated in the REPL, the first put succeeded (presumably 
because it went on the buffer), but subsequent puts blocked:

(def c (async/chan 1))
(def m (async/mult c))
(def out (async/chan (async/sliding-buffer 3)))
(async/tap m out)
(>!! c true) ;; succeeds
(>!! c true) ;; blocks forever

This leads me to wonder if core.async itself somehow got into a bad state. 
It's entirely possible I caused this by misusing the API somewhere in the 
codebase, but we use core.async so extensively that I wouldn't know where 
to begin looking.

I'm wondering if someone more familiar with core.async internals has an 
idea about what could cause the above situation. Or if we notice it 
happening again, what could I do to gather more helpful information.

I also have a redacted thread dump, in case it's useful:

https://gist.github.com/anonymous/65049ffdd37d43df8f23630928e8fed0

Any help would be much appreciated,

Aaron

P.S. core.async has been a godsend in terms of helping us structure and 
modularize our large system.  Thank you to all those who contributed to 
this wonderful library!

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: core.async got in a bad state?

[Alex Miller]

go blocks are multiplexed over a thread pool which has (by default) 8 
threads. You should never perform any kind of blocking activity inside a go 
block, because if every go block in work happens to end up blocked, you 
will prevent all go blocks from making any further progress. It sounds to 
me like that's what has happened here. The go block threads are named 
"async-dispatch-" and it looks like there are 8 blocked ones in your 
thread dump.

It also looks like they are all blocking on a >!!, which is a blocking 
call. So I would look for a go block that contains a >!! and convert that 
to a >! or do something else to avoid blocking there.


On Tuesday, August 29, 2017 at 11:48:25 AM UTC-5, Aaron Iba wrote:
>
> My company has a production system that uses core.async extensively. We've 
> been running it 24/7 for over a year with occasional restarts to update 
> things and add features, and so far core.async has been working great.
>
> The other day, during a particularly high workload, the whole system got 
> locked up. All the channels seemed blocked at once.  I was able to connect 
> with a REPL and poke around, and noticed strange behavior of core.async. 
> Specifically, the following code, when evaluated in the REPL, blocked on 
> the put (third expression):
>
> (def c (async/chan))
> (go-loop []
>   (when-some [x ( (println x)
> (recur)))
> (>!! c true)
>
> Whereas on any fresh system, the above expressions obviously succeed.
>
> Puts succeeded if they went onto the channel's buffer, but not when they 
> should go through to a consumer. For example with the following 
> expressions, evaluated in the REPL, the first put succeeded (presumably 
> because it went on the buffer), but subsequent puts blocked:
>
> (def c (async/chan 1))
> (def m (async/mult c))
> (def out (async/chan (async/sliding-buffer 3)))
> (async/tap m out)
> (>!! c true) ;; succeeds
> (>!! c true) ;; blocks forever
>
> This leads me to wonder if core.async itself somehow got into a bad state. 
> It's entirely possible I caused this by misusing the API somewhere in the 
> codebase, but we use core.async so extensively that I wouldn't know where 
> to begin looking.
>
> I'm wondering if someone more familiar with core.async internals has an 
> idea about what could cause the above situation. Or if we notice it 
> happening again, what could I do to gather more helpful information.
>
> I also have a redacted thread dump, in case it's useful:
>
> https://gist.github.com/anonymous/65049ffdd37d43df8f23630928e8fed0
>
> Any help would be much appreciated,
>
> Aaron
>
> P.S. core.async has been a godsend in terms of helping us structure and 
> modularize our large system.  Thank you to all those who contributed to 
> this wonderful library!
>
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: core.async got in a bad state?

[Timothy Baldridge]

To add to what Alex said, look at this trace:
https://gist.github.com/anonymous/65049ffdd37d43df8f23630928e8fed0#file-thread-dump-out-L1337-L1372

Here we see a go block calling mapcat, and inside the inner map something
is calling >!!. As Alex mentioned this can be a source of deadlocks. No
code called by a go block should ever call the blocking variants of
core.async functions (!!, alts!!, etc.). So I'd start at the code
redacted in those lines and go from there.



On Tue, Aug 29, 2017 at 11:09 AM, Alex Miller  wrote:

> go blocks are multiplexed over a thread pool which has (by default) 8
> threads. You should never perform any kind of blocking activity inside a go
> block, because if every go block in work happens to end up blocked, you
> will prevent all go blocks from making any further progress. It sounds to
> me like that's what has happened here. The go block threads are named
> "async-dispatch-" and it looks like there are 8 blocked ones in your
> thread dump.
>
> It also looks like they are all blocking on a >!!, which is a blocking
> call. So I would look for a go block that contains a >!! and convert that
> to a >! or do something else to avoid blocking there.
>
>
> On Tuesday, August 29, 2017 at 11:48:25 AM UTC-5, Aaron Iba wrote:
>>
>> My company has a production system that uses core.async extensively.
>> We've been running it 24/7 for over a year with occasional restarts to
>> update things and add features, and so far core.async has been working
>> great.
>>
>> The other day, during a particularly high workload, the whole system got
>> locked up. All the channels seemed blocked at once.  I was able to connect
>> with a REPL and poke around, and noticed strange behavior of core.async.
>> Specifically, the following code, when evaluated in the REPL, blocked on
>> the put (third expression):
>>
>> (def c (async/chan))
>> (go-loop []
>>   (when-some [x (> (println x)
>> (recur)))
>> (>!! c true)
>>
>> Whereas on any fresh system, the above expressions obviously succeed.
>>
>> Puts succeeded if they went onto the channel's buffer, but not when they
>> should go through to a consumer. For example with the following
>> expressions, evaluated in the REPL, the first put succeeded (presumably
>> because it went on the buffer), but subsequent puts blocked:
>>
>> (def c (async/chan 1))
>> (def m (async/mult c))
>> (def out (async/chan (async/sliding-buffer 3)))
>> (async/tap m out)
>> (>!! c true) ;; succeeds
>> (>!! c true) ;; blocks forever
>>
>> This leads me to wonder if core.async itself somehow got into a bad
>> state. It's entirely possible I caused this by misusing the API somewhere
>> in the codebase, but we use core.async so extensively that I wouldn't know
>> where to begin looking.
>>
>> I'm wondering if someone more familiar with core.async internals has an
>> idea about what could cause the above situation. Or if we notice it
>> happening again, what could I do to gather more helpful information.
>>
>> I also have a redacted thread dump, in case it's useful:
>>
>> https://gist.github.com/anonymous/65049ffdd37d43df8f23630928e8fed0
>>
>> Any help would be much appreciated,
>>
>> Aaron
>>
>> P.S. core.async has been a godsend in terms of helping us structure and
>> modularize our large system.  Thank you to all those who contributed to
>> this wonderful library!
>>
>> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>



-- 
“One of the main causes of the fall of the Roman Empire was that–lacking
zero–they had no way to indicate successful termination of their C
programs.”
(Robert Firth)

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: core.async got in a bad state?

[Gary Trakhman]

Hm, I came across a similar ordering invariant (No code called by a go
block should ever call the blocking variants of core.async functions) while
wrapping an imperative API, and I thought it might be useful to use
vars/binding to enforce it.  Has this or other approaches been considered
in core.async?  I could see a *fixed-thread-pool* var being set and >!!
checking for false.

An analogy in existing clojure.core would be the STM commute's 'must be
running in a transaction' check that uses a threadlocal.
https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/LockingTransaction.java#L205

On Tue, Aug 29, 2017 at 1:30 PM Timothy Baldridge 
wrote:

> To add to what Alex said, look at this trace:
> https://gist.github.com/anonymous/65049ffdd37d43df8f23630928e8fed0#file-thread-dump-out-L1337-L1372
>
> Here we see a go block calling mapcat, and inside the inner map something
> is calling >!!. As Alex mentioned this can be a source of deadlocks. No
> code called by a go block should ever call the blocking variants of
> core.async functions (!!, alts!!, etc.). So I'd start at the code
> redacted in those lines and go from there.
>
>
>
> On Tue, Aug 29, 2017 at 11:09 AM, Alex Miller  wrote:
>
>> go blocks are multiplexed over a thread pool which has (by default) 8
>> threads. You should never perform any kind of blocking activity inside a go
>> block, because if every go block in work happens to end up blocked, you
>> will prevent all go blocks from making any further progress. It sounds to
>> me like that's what has happened here. The go block threads are named
>> "async-dispatch-" and it looks like there are 8 blocked ones in your
>> thread dump.
>>
>> It also looks like they are all blocking on a >!!, which is a blocking
>> call. So I would look for a go block that contains a >!! and convert that
>> to a >! or do something else to avoid blocking there.
>>
>>
>> On Tuesday, August 29, 2017 at 11:48:25 AM UTC-5, Aaron Iba wrote:
>>>
>>> My company has a production system that uses core.async extensively.
>>> We've been running it 24/7 for over a year with occasional restarts to
>>> update things and add features, and so far core.async has been working
>>> great.
>>>
>>> The other day, during a particularly high workload, the whole system got
>>> locked up. All the channels seemed blocked at once.  I was able to connect
>>> with a REPL and poke around, and noticed strange behavior of core.async.
>>> Specifically, the following code, when evaluated in the REPL, blocked on
>>> the put (third expression):
>>>
>>> (def c (async/chan))
>>> (go-loop []
>>>   (when-some [x (>> (println x)
>>> (recur)))
>>> (>!! c true)
>>>
>>> Whereas on any fresh system, the above expressions obviously succeed.
>>>
>>> Puts succeeded if they went onto the channel's buffer, but not when they
>>> should go through to a consumer. For example with the following
>>> expressions, evaluated in the REPL, the first put succeeded (presumably
>>> because it went on the buffer), but subsequent puts blocked:
>>>
>>> (def c (async/chan 1))
>>> (def m (async/mult c))
>>> (def out (async/chan (async/sliding-buffer 3)))
>>> (async/tap m out)
>>> (>!! c true) ;; succeeds
>>> (>!! c true) ;; blocks forever
>>>
>>> This leads me to wonder if core.async itself somehow got into a bad
>>> state. It's entirely possible I caused this by misusing the API somewhere
>>> in the codebase, but we use core.async so extensively that I wouldn't know
>>> where to begin looking.
>>>
>>> I'm wondering if someone more familiar with core.async internals has an
>>> idea about what could cause the above situation. Or if we notice it
>>> happening again, what could I do to gather more helpful information.
>>>
>>> I also have a redacted thread dump, in case it's useful:
>>>
>>> https://gist.github.com/anonymous/65049ffdd37d43df8f23630928e8fed0
>>>
>>> Any help would be much appreciated,
>>>
>>> Aaron
>>>
>>> P.S. core.async has been a godsend in terms of helping us structure and
>>> modularize our large system.  Thank you to all those who contributed to
>>> this wonderful library!
>>>
>>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clojure@googlegroups.com
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+unsubscr...@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to clojure+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> --
> “One of the main causes of the fall of the Roman Empire was that–lacking
> zero–they had no way to indicate successful termination of thei

Re: core.async got in a bad state?

[Aaron Iba]

Ahh that makes a lot of sense.  Indeed, I'm guilty of doing a blocking >!! 
inside a go-block.  I was so careful to avoid other kinds of blocking calls 
(like IO) that I forgot that blocking variants of core.async calls 
themselves were forbidden.

Thank you for pointing this out!  I will rewire things to not do this.

Per Gary's suggestion, I also think it'd be useful if core.async blocking 
ops checked a dynamic var (or a property of the thread itself) and at least 
warned if they are being called from a forbidden context.  To resolve my 
original issue, I'm considering doing this in my dev environment:

(doseq [v '[!!]]
  (alter-var-root (ns-resolve 'clojure.core.async v)
  (fn [f]
(fn [& args]
  (if (.startsWith (.getName (Thread/currentThread))
   "async-dispatch-")
(throw (Exception. (str v " called inside 
async-dispatch")))
(apply f args))




On Tuesday, August 29, 2017 at 1:43:53 PM UTC-4, Gary Trakhman wrote:
>
> Hm, I came across a similar ordering invariant (No code called by a go 
> block should ever call the blocking variants of core.async functions) while 
> wrapping an imperative API, and I thought it might be useful to use 
> vars/binding to enforce it.  Has this or other approaches been considered 
> in core.async?  I could see a *fixed-thread-pool* var being set and >!! 
> checking for false.
>
> An analogy in existing clojure.core would be the STM commute's 'must be 
> running in a transaction' check that uses a threadlocal. 
> https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/LockingTransaction.java#L205
>
> On Tue, Aug 29, 2017 at 1:30 PM Timothy Baldridge  > wrote:
>
>> To add to what Alex said, look at this trace: 
>> https://gist.github.com/anonymous/65049ffdd37d43df8f23630928e8fed0#file-thread-dump-out-L1337-L1372
>>
>> Here we see a go block calling mapcat, and inside the inner map something 
>> is calling >!!. As Alex mentioned this can be a source of deadlocks. No 
>> code called by a go block should ever call the blocking variants of 
>> core.async functions (!!, alts!!, etc.). So I'd start at the code 
>> redacted in those lines and go from there. 
>>
>>
>>
>> On Tue, Aug 29, 2017 at 11:09 AM, Alex Miller > > wrote:
>>
>>> go blocks are multiplexed over a thread pool which has (by default) 8 
>>> threads. You should never perform any kind of blocking activity inside a go 
>>> block, because if every go block in work happens to end up blocked, you 
>>> will prevent all go blocks from making any further progress. It sounds to 
>>> me like that's what has happened here. The go block threads are named 
>>> "async-dispatch-" and it looks like there are 8 blocked ones in your 
>>> thread dump.
>>>
>>> It also looks like they are all blocking on a >!!, which is a blocking 
>>> call. So I would look for a go block that contains a >!! and convert that 
>>> to a >! or do something else to avoid blocking there.
>>>
>>>
>>> On Tuesday, August 29, 2017 at 11:48:25 AM UTC-5, Aaron Iba wrote:

 My company has a production system that uses core.async extensively. 
 We've been running it 24/7 for over a year with occasional restarts to 
 update things and add features, and so far core.async has been working 
 great.

 The other day, during a particularly high workload, the whole system 
 got locked up. All the channels seemed blocked at once.  I was able to 
 connect with a REPL and poke around, and noticed strange behavior of 
 core.async. Specifically, the following code, when evaluated in the REPL, 
 blocked on the put (third expression):

 (def c (async/chan))
 (go-loop []
   (when-some [x (>>> (println x)
 (recur)))
 (>!! c true)

 Whereas on any fresh system, the above expressions obviously succeed.

 Puts succeeded if they went onto the channel's buffer, but not when 
 they should go through to a consumer. For example with the following 
 expressions, evaluated in the REPL, the first put succeeded (presumably 
 because it went on the buffer), but subsequent puts blocked:

 (def c (async/chan 1))
 (def m (async/mult c))
 (def out (async/chan (async/sliding-buffer 3)))
 (async/tap m out)
 (>!! c true) ;; succeeds
 (>!! c true) ;; blocks forever

 This leads me to wonder if core.async itself somehow got into a bad 
 state. It's entirely possible I caused this by misusing the API somewhere 
 in the codebase, but we use core.async so extensively that I wouldn't know 
 where to begin looking.

 I'm wondering if someone more familiar with core.async internals has an 
 idea about what could cause the above situation. Or if we notice it 
 happening again, what could I do to gather more helpful information.

 I also have a redacted thread dump, in case it's usefu

Re: core.async got in a bad state?

[Alex Miller]

We did actually discuss doing something like this a long time ago. The 
worry that comes to mind is whether it should actually be forbidden (an 
invariant) or merely strong frowned upon. It is possible to arrange a 
situation where you know (based on other knowledge) that a put on a channel 
will succeed. Should that be allowed? I'd have to hammock on that one for a 
while.

On Tuesday, August 29, 2017 at 3:19:51 PM UTC-5, Aaron Iba wrote:
>
> Ahh that makes a lot of sense.  Indeed, I'm guilty of doing a blocking >!! 
> inside a go-block.  I was so careful to avoid other kinds of blocking calls 
> (like IO) that I forgot that blocking variants of core.async calls 
> themselves were forbidden.
>
> Thank you for pointing this out!  I will rewire things to not do this.
>
> Per Gary's suggestion, I also think it'd be useful if core.async blocking 
> ops checked a dynamic var (or a property of the thread itself) and at least 
> warned if they are being called from a forbidden context.  To resolve my 
> original issue, I'm considering doing this in my dev environment:
>
> (doseq [v '[!!]]
>   (alter-var-root (ns-resolve 'clojure.core.async v)
>   (fn [f]
> (fn [& args]
>   (if (.startsWith (.getName (Thread/currentThread))
>"async-dispatch-")
> (throw (Exception. (str v " called inside 
> async-dispatch")))
> (apply f args))
>
>
>
>
> On Tuesday, August 29, 2017 at 1:43:53 PM UTC-4, Gary Trakhman wrote:
>>
>> Hm, I came across a similar ordering invariant (No code called by a go 
>> block should ever call the blocking variants of core.async functions) while 
>> wrapping an imperative API, and I thought it might be useful to use 
>> vars/binding to enforce it.  Has this or other approaches been considered 
>> in core.async?  I could see a *fixed-thread-pool* var being set and >!! 
>> checking for false.
>>
>> An analogy in existing clojure.core would be the STM commute's 'must be 
>> running in a transaction' check that uses a threadlocal. 
>> https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/LockingTransaction.java#L205
>>
>> On Tue, Aug 29, 2017 at 1:30 PM Timothy Baldridge  
>> wrote:
>>
>>> To add to what Alex said, look at this trace: 
>>> https://gist.github.com/anonymous/65049ffdd37d43df8f23630928e8fed0#file-thread-dump-out-L1337-L1372
>>>
>>> Here we see a go block calling mapcat, and inside the inner map 
>>> something is calling >!!. As Alex mentioned this can be a source of 
>>> deadlocks. No code called by a go block should ever call the blocking 
>>> variants of core.async functions (!!, alts!!, etc.). So I'd start at 
>>> the code redacted in those lines and go from there. 
>>>
>>>
>>>
>>> On Tue, Aug 29, 2017 at 11:09 AM, Alex Miller  
>>> wrote:
>>>
 go blocks are multiplexed over a thread pool which has (by default) 8 
 threads. You should never perform any kind of blocking activity inside a 
 go 
 block, because if every go block in work happens to end up blocked, you 
 will prevent all go blocks from making any further progress. It sounds to 
 me like that's what has happened here. The go block threads are named 
 "async-dispatch-" and it looks like there are 8 blocked ones in your 
 thread dump.

 It also looks like they are all blocking on a >!!, which is a blocking 
 call. So I would look for a go block that contains a >!! and convert that 
 to a >! or do something else to avoid blocking there.


 On Tuesday, August 29, 2017 at 11:48:25 AM UTC-5, Aaron Iba wrote:
>
> My company has a production system that uses core.async extensively. 
> We've been running it 24/7 for over a year with occasional restarts to 
> update things and add features, and so far core.async has been working 
> great.
>
> The other day, during a particularly high workload, the whole system 
> got locked up. All the channels seemed blocked at once.  I was able to 
> connect with a REPL and poke around, and noticed strange behavior of 
> core.async. Specifically, the following code, when evaluated in the REPL, 
> blocked on the put (third expression):
>
> (def c (async/chan))
> (go-loop []
>   (when-some [x ( (println x)
> (recur)))
> (>!! c true)
>
> Whereas on any fresh system, the above expressions obviously succeed.
>
> Puts succeeded if they went onto the channel's buffer, but not when 
> they should go through to a consumer. For example with the following 
> expressions, evaluated in the REPL, the first put succeeded (presumably 
> because it went on the buffer), but subsequent puts blocked:
>
> (def c (async/chan 1))
> (def m (async/mult c))
> (def out (async/chan (async/sliding-buffer 3)))
> (async/tap m out)
> (>!! c true) ;; succeeds
> (>!! c true) ;; block

Re: Sum types in Clojure? Better to represent as tagged records or as variant vectors?

[Didier]

Guten-tag looks pretty great. I think your readme also explains the 
differences pretty well.

I definitely feel like variants are really not great as data-structures to 
manipulate, but much better as declarative data constructors. Your library 
kinda unifies both into one type, its pretty clever.

On Monday, 28 August 2017 13:53:33 UTC-7, Reid McKenzie wrote:
>
> FWIW I wrote a library around defining tagged map types 
> https://github.com/arrdem/guten-tag and used it heavily in the Grimoire 
> implementation almost entirely to my satisfaction. No idea if anyone else 
> ever picked it up, and due to implementation details of Clojure's let map 
> destructuring you can't just treat one of my map wrapper types as a normal 
> Clojure map and :keys it apart but it works great with core.match.
>
> Reid
>
> On Friday, August 25, 2017 at 1:30:37 PM UTC-7, tbc++ wrote:
>>
>> >> they're a little nicer to type and read
>>
>> And that's where I have to disagree. The problem with most of these 
>> options is they complect ordering with meaning. 
>>
>> [:image/file "/images/image" :jpeg]
>>
>> Here I have no idea what these values mean. I have to have out-of-band 
>> information about what offset in the vector corresponds to what value. 
>> Functions have this same problem, look no further than `cons` vs `conj` to 
>> see potential confusion on argument ordering. 
>>
>> So why don't we only use maps for function arguments? Well mostly to make 
>> the functions easier to manipulate by humans. But some of the best 
>> libraries for web APIs (AWS) that I've used have used almost an exclusively 
>> map based interface. 
>>
>> Once again I have to repeat my mantra about DSLs. Don't start with DSLs. 
>> Start with maps. Build everything off of maps. They're extensible, easy to 
>> introspect, and can drive a host of metaprogramming algorithms. If maps are 
>> to hard to understand, build constructor functions on top of them. Then 
>> finally build a DSL on top, if you need it. 
>>
>> Frankly, I have so many things I have to remember during programming. I'd 
>> much rather see a very uniform map-based interface. Than any sort of nested 
>> vectors, tagged values, or anything else. 
>>
>> Surely we can't say that this:
>> >> [[:image/file :image/web] :image.file/path "/images/image" 
>> :image.file/format :jpeg :image.web/url "www.myimage.com/image"]
>>
>> Is a better interface than:
>>
>> {:image.file/path "/images/image"
>>  :image.file/format :jpeg
>>  :image.web/url "www.myimage.com/image"}
>>
>> And as I said before, spec is designed from the start to support data in 
>> this format. Stating "this is a file", "this is a web image". Is just book 
>> keeping that doesn't need to be done. Is a map a :image/web? Well check its 
>> members and see if they match the spec. If they match, then you have a 
>> :image/web. No need for sum types, tags, wrapping values in vectors. Simply 
>> state what a thing should have for it to conform to an interface, and 
>> forget whatever else might be in there. It couldn't be simpler. 
>>
>> On Fri, Aug 25, 2017 at 11:59 AM, Didier  wrote:
>>
>>> Thanks for everyone's great input.
>>>
>>> Currently, I see the big distinction being concise vs extension. Maybe 
>>> for streaming variants would be better as the format would be smaller to 
>>> transfer over a wire. And for small short lived programs, or things you 
>>> know won't need extension, variants offer a slightly more convenient 
>>> structure to work with.
>>>
>>> I think both can be specced easily. S/or a few s/tuple to spec a 
>>> variant. And multi-spec over a set for the map version.
>>>
>>> I'd like to explore then the issue of extensibility with variants. How 
>>> would you extend them, is there really no way? This is some of my 
>>> brainstorming thoughts.
>>>
>>> 1) How to design a variant of more then one value?
>>>
>>>
>>> 1.1)
>>>
>>> I know this is a standard variant of one value:
>>>
>>> [:image/web "www.myimage.com/image.jpg"]
>>>
>>> I believe to extend it to more values, you would do:
>>>
>>> [:image/file "/images/image" :jpeg]
>>>
>>> That is, you'd threat it as a constructor function, which creates an 
>>> :image/file type and takes an ordered list of arguments. This way, each 
>>> variant type can even be overloaded on their arguments the way you'd 
>>> overload a function.
>>>
>>> [:image/file "/images/image"]
>>>
>>> Can default to format :png for example, when using the one arg 
>>> constructor.
>>>
>>> 1.2)
>>>
>>> An alternative is to keep variants as vector pairs, and nest them.
>>>
>>> [:image/file [:image/file-path "/images/image"] 
>>> [:image/file-format:jpeg]]
>>>
>>> In this form, variants are more like their map counterpart. Each element 
>>> is named and itself a variant.
>>>
>>> 1.3)
>>>
>>> You could also decide to limit variants to strict pairs, so the second 
>>> element of any variant is either a variant or a vector of variants.
>>>
>>> [:image/file [[:image/file-path "/images/image"] 
>>> [:image/f

Re: core.async got in a bad state?

[Didier]

>
> No code called by a go block should ever call the blocking variants of 
> core.async functions (!!, alts!!, etc.). So I'd start at the code 
> redacted in those lines and go from there.
>

Seems like a good use case for a static code analyser.  Maybe a 
contribution to https://github.com/jonase/eastwood

On Tuesday, 29 August 2017 14:40:08 UTC-7, Alex Miller wrote:
>
> We did actually discuss doing something like this a long time ago. The 
> worry that comes to mind is whether it should actually be forbidden (an 
> invariant) or merely strong frowned upon. It is possible to arrange a 
> situation where you know (based on other knowledge) that a put on a channel 
> will succeed. Should that be allowed? I'd have to hammock on that one for a 
> while.
>
> On Tuesday, August 29, 2017 at 3:19:51 PM UTC-5, Aaron Iba wrote:
>>
>> Ahh that makes a lot of sense.  Indeed, I'm guilty of doing a blocking 
>> >!! inside a go-block.  I was so careful to avoid other kinds of blocking 
>> calls (like IO) that I forgot that blocking variants of core.async calls 
>> themselves were forbidden.
>>
>> Thank you for pointing this out!  I will rewire things to not do this.
>>
>> Per Gary's suggestion, I also think it'd be useful if core.async blocking 
>> ops checked a dynamic var (or a property of the thread itself) and at least 
>> warned if they are being called from a forbidden context.  To resolve my 
>> original issue, I'm considering doing this in my dev environment:
>>
>> (doseq [v '[!!]]
>>   (alter-var-root (ns-resolve 'clojure.core.async v)
>>   (fn [f]
>> (fn [& args]
>>   (if (.startsWith (.getName (Thread/currentThread))
>>"async-dispatch-")
>> (throw (Exception. (str v " called inside 
>> async-dispatch")))
>> (apply f args))
>>
>>
>>
>>
>> On Tuesday, August 29, 2017 at 1:43:53 PM UTC-4, Gary Trakhman wrote:
>>>
>>> Hm, I came across a similar ordering invariant (No code called by a go 
>>> block should ever call the blocking variants of core.async functions) while 
>>> wrapping an imperative API, and I thought it might be useful to use 
>>> vars/binding to enforce it.  Has this or other approaches been considered 
>>> in core.async?  I could see a *fixed-thread-pool* var being set and >!! 
>>> checking for false.
>>>
>>> An analogy in existing clojure.core would be the STM commute's 'must be 
>>> running in a transaction' check that uses a threadlocal. 
>>> https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/LockingTransaction.java#L205
>>>
>>> On Tue, Aug 29, 2017 at 1:30 PM Timothy Baldridge  
>>> wrote:
>>>
 To add to what Alex said, look at this trace: 
 https://gist.github.com/anonymous/65049ffdd37d43df8f23630928e8fed0#file-thread-dump-out-L1337-L1372

 Here we see a go block calling mapcat, and inside the inner map 
 something is calling >!!. As Alex mentioned this can be a source of 
 deadlocks. No code called by a go block should ever call the blocking 
 variants of core.async functions (!!, alts!!, etc.). So I'd start at 
 the code redacted in those lines and go from there. 



 On Tue, Aug 29, 2017 at 11:09 AM, Alex Miller  
 wrote:

> go blocks are multiplexed over a thread pool which has (by default) 8 
> threads. You should never perform any kind of blocking activity inside a 
> go 
> block, because if every go block in work happens to end up blocked, you 
> will prevent all go blocks from making any further progress. It sounds to 
> me like that's what has happened here. The go block threads are named 
> "async-dispatch-" and it looks like there are 8 blocked ones in your 
> thread dump.
>
> It also looks like they are all blocking on a >!!, which is a blocking 
> call. So I would look for a go block that contains a >!! and convert that 
> to a >! or do something else to avoid blocking there.
>
>
> On Tuesday, August 29, 2017 at 11:48:25 AM UTC-5, Aaron Iba wrote:
>>
>> My company has a production system that uses core.async extensively. 
>> We've been running it 24/7 for over a year with occasional restarts to 
>> update things and add features, and so far core.async has been working 
>> great.
>>
>> The other day, during a particularly high workload, the whole system 
>> got locked up. All the channels seemed blocked at once.  I was able to 
>> connect with a REPL and poke around, and noticed strange behavior of 
>> core.async. Specifically, the following code, when evaluated in the 
>> REPL, 
>> blocked on the put (third expression):
>>
>> (def c (async/chan))
>> (go-loop []
>>   (when-some [x (> (println x)
>> (recur)))
>> (>!! c true)
>>
>> Whereas on any fresh system, the above expressions obviously succeed.
>>
>>