I had some time to play with core.async further today, and I learned some
more about it.
First let me say that I've completed the implementation of core.async on
top of Pulsar (I've copied the alt! macro from the core.async repository
and placed a copyright notice around it). I ran some of core.async tests on
it, and they pass unchanged. But before I get back to what I've found, I
want to address your remarks, Tim. My main point wasn't performance in
general, but performance in a distributed system, and I may have expressed
myself too mildly.
Using alts! with distributed channels is not just slow - it is extremely
difficult to do right, and actually makes it harder to build a robust
system (to go back to Joe Armstrong). I might very well be wrong, but a
distributed alts! would translate, in all but few cases, to a distributed
transaction. And distributed transactions are hard.
Now, this is what I've found playing with core.async today. First, the API
is, unsurprisingly considering its source, simply brilliant. It's a paragon
of simplicity (although I do forget the meaning of >! and <!; I mean, which
is send and which is receive? then I remember, oh, the ! stands for the
channel, but again I forget which side of the bang the arrow goes). The
alts! statement is a thing of beauty, although, I do stand by my opinion
that it's too general by far.
With regards to performance: core.async performs quite well. On several
small benchmarks it outperforms Pulsar's core.async, like on this one (by
Alex Miller):
(let [n 1000
cs (repeatedly n chan)
begin (System/currentTimeMillis)]
(doseq [c cs] (go (>! c "hi")))
(dotimes [i n]
(let [[v c] (alts!! cs)]
(assert (= "hi" v))))
(println "Read" n "msgs in" (- (System/currentTimeMillis) begin) "ms"))
When I investigated, I found the cause to be (mostly) Pulsar's use of FJPool
vs. core.async's use of ThreadPoolExecutor. In the above example, the go-blocks
(we need a better name for them; I'm used to calling them fibers in Pulsar, so
maybe fibers? or gos?) don't block. They are just all put into the thread-pool
once, where they execute to completion. TPE beats FJP in that scenario. FJP
shines when its tasks keep spawning other tasks, which translates to gos
parking and unparking other gos.
So when you have lots of interaction between fibers, that's when Pulsar's async
beats core.async. Like in this example:
(let [n 10000
cs (repeatedly n chan) ; or; #(chan 10)) ;
begin (System/currentTimeMillis)]
(doseq [c cs] (go (dotimes [i 100] (>! c i))))
(let [cs2 (doall (map #(go
(loop [i 0 sum (int 0)]
(if (< i 100)
(recur (inc i) (int (+ sum (<! %))))
sum)))
cs))]
(doseq [c cs2]
(let [v (<!! c)]
(assert (= 4950 v))))
(println "Read" n "msgs in" (- (System/currentTimeMillis) begin) "ms")))
Also, when benchmarking core.async on Pulsar (or any program using Pulsar)
it is *extremely* important to run "lein trampoline". Without trampoline,
Pulsar runs *much* slower. core.async isn't affected much by the lack of
trampoline.
In any case, we now have two, quite different, implementations of the
core.async API. I'd be interested to learn what other people find.
To try core.async on Pulsar, you need the dev release:
[co.paralleluniverse/pulsar "0.2-SNAPSHOT"]
On Monday, July 1, 2013 6:46:26 PM UTC+3, tbc++ wrote:
>
> Ben, you make a good point. I guess I was stating was something more along
> the lines of the following. We have really really good concurrency
> primitives (atom, ref, agent) and I think it would be a mistake to reach
> for channels when one of those would do.
>
> core.async is pretty fast, but reducers will probably work faster for
> normal map/reduce needs. You can write something like an agent using
> channels, but it won't be as fast as Clojure's agents.
>
> So I should have stated this much clearer "to focus a system purely on
> core.async would be to ignore the other features of Clojure. Use core.async
> when other core language features won't work"
>
> Timothy
>
>
> On Mon, Jul 1, 2013 at 9:27 AM, Ben Wolfson <wol...@gmail.com<javascript:>
> > wrote:
>
>> On Mon, Jul 1, 2013 at 7:10 AM, Timothy Baldridge
>> <tbald...@gmail.com<javascript:>
>> > wrote:
>>
>>> It's my opinion that core.async shouldn't be used a concurrency
>>> mechanism, or as a new programming paradigm. Instead, it should be used as
>>> a communication method between sequential processes, and as a de-coupling
>>> method between modules. Let's go back to Rich's original post on the
>>> library:
>>>
>>> "There comes a time in all good programs when components or subsystems
>>> must stop communicating directly with one another. This is often achieved
>>> via the introduction of queues between the producers of data and the
>>> consumers/processors of that data."
>>>
>>
>> Well, if you're looking for guidance as to how the library should be used
>> by reading tea leaves in a blog post introducing them, compare:
>>
>> "Note that, unlike other Clojure concurrency constructs, channels, like
>> all communications, are subject to deadlocks, the simplest being waiting
>> for a message that will never arrive, which must be dealt with manually via
>> timeouts etc."
>>
>> "Unlike other ... concurrency constructs" == "this is also a concurrency
>> construct". It's possible, I suppose, to maintain that it's a concurrency
>> construct that shouldn't be used for concurrency-in-general, but that
>> strikes me as a losing proposition; it *is* a concurrency construct, and
>> nothing about the library itself requires that it be used the way you're
>> suggesting. (It would be nice if there were some guarantee of linearity
>> for channels so that I don't try to pass one to e.g. take-while and then
>> also try to read from it elsewhere, but that's not directly related.) Also,
>> if---as the references to go and Erlang suggest---it will be possible to
>> have hundreds of thousands of go blocks talking over channels to each
>> other, performance considerations seem pretty reasonable.
>>
>> --
>> Ben Wolfson
>> "Human kind has used its intelligence to vary the flavour of drinks,
>> which may be sweet, aromatic, fermented or spirit-based. ... Family and
>> social life also offer numerous other occasions to consume drinks for
>> pleasure." [Larousse, "Drink" entry]
>>
>> --
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clo...@googlegroups.com<javascript:>
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+u...@googlegroups.com <javascript:>
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to clojure+u...@googlegroups.com <javascript:>.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>>
>>
>
>
>
> --
> “One of the main causes of the fall of the Roman Empire was that–lacking
> zero–they had no way to indicate successful termination of their C
> programs.”
> (Robert Firth)
>
--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
