Leon: your :useless example is equivalent to one using the hypothetical
peek operation and to the ack-channel solution, but what you define as
":useless multithreading" because at one time only one thread is working,
is a valid synchronization scenario and the one I was searching for. The
objective here was not to max CPU usage with these two threads / go-blocks.
As I said in a previous mail "maybe you want to reserve the cpu cycles for
other tasks, or the producer needs a confirmation for other purposes before
computing or requesting the value from another party, this is a distilled
example". Also think the producer after sending the computed value and
before sending the :useless signal can do other things, maybe contacting
other services to pre-fetch data used in the next value.
Also the :useless example needs more code on the producer side than a
simple (>! c v). Given there is no peek operation and the producer code
must participate explicitly in the synchronization, I prefer the ack
solution over the :useless one.
You are right about impossibility of using alts! in the producer code to
check if the chan can be written to before computing the value. I forgot
alts! is a function, not a macro, and you can't pass expressions to it to
be evaluated when the chan becomes available to write.
Your last sample seems an useful way to manage simple stateless
single-value producers from the consumer, but producers can be long-running
go-blocks with their own state and with channels to other services, so I
think is a little off base the original question.
Saludos,
Nahuel Greco.
On Thu, Oct 9, 2014 at 4:16 PM, Leon Grapenthin <grapenthinl...@gmail.com>
wrote:
> On Thursday, October 9, 2014 2:22:50 PM UTC+2, Nahuel Greco wrote:
>>
>> Fluid: as you said, backpressure on the outgoing channel will be
>> sufficient to avoid producer to overrun consumer (without using any extra
>> ack-channel), but the producer will compute a new not-yet-used value before
>> blocking on sending it to consumer. That's what I want to avoid.
>>
>
> You can avoid this very easily. Everytime before the producer expensively
> calculates its next value to put it on the channel, make it put a cheaply
> calculated value, e. g. the keyword :useless, on the channel instead.
>
> Now on the consumer side you can do this:
>
> 0. Blocking (immediate take): Consume the keyword :useless = The producer
> will begin production
> 1. *This is what you want from peek! *Blocking take = Waiting for the
> producer to have produced the expensive value, consume it *but the
> producer won't calculate the next value (because he puts the :useless
> keyword which won't be consumed until later)*
> 2. Wait for external service to be ready, put value there
> 3. Go to step 1 trigger production of the next value
>
> Assuming the producer and the consumer run on different threads, what have
> you won by introducing step 0?
>
> 1. The producer thread can't begin to work until the consumer thread did
> his work
> 2. The consumer thread can't begin to work until the producer thread did
> his work
>
> = At every point in time only one thread performs. Entirely :useless
> multi-threading.
>
> I can also use alts! on the producer to check if he can write to the
>> channel before computing the value, [...]
>>
> You can't
>
>
>
>> [...]
>>
>
>
>> Fregal: If middleman is managing all the communication with the external
>> service for *multiple* producers, then it can't be reduced to a simple
>> synchronous function call from a producer, hence the necessity of
>> core.async (the alternative is to descend to locks world).
>>
>
> If I understand you right, you would want to start production of multiple
> values in parallel, wait for the first to be ready, then put that to the
> external service and loop. Here core.async can be of help. E. g.
> (defn exp-calcs
> []
> #{(a/thread calc-exp-val-1)
> (a/thread calc-exp-val-2)
> (a/thread calc-exp-val-3)
> (a/thread calc-exp-val-4)
> (a/thread calc-exp-val-5)
> ; ...
> })
>
> (loop [chans (exp-calcs)]
> (let [[val ch] (a/alts!! (seq exp-calcs))]
> ;; wait for external service, put val there
>
> ;; also have the cancellation ch in chans and use it to guard the
> ;; following form
> (let [chans (disj chans ch)]
> (recur (if (seq chans)
> chans
> (exp-calcs))))))
>
>
>
> Again, no peek required.
>
>
>>
>>
>>
>> Saludos,
>> Nahuel Greco.
>>
>> On Thu, Oct 9, 2014 at 9:07 AM, Fergal Byrne <fergalby...@gmail.com>
>> wrote:
>>
>>> Hi Nahuel,
>>>
>>> I think it's worth stepping back from discussing solutions and have a
>>> look at the problem. If I'm reading things right you need the following:
>>>
>>> 1. Producer produces values one at a time - should not step until last
>>> value has been handled correctly elsewhere.
>>> 2. Middleman needs to consume one value and (potentially) retry an
>>> external service until it can verify correct handling.
>>> 3. Producer should not interpret the consumption of a value as a signal
>>> to proceed.
>>>
>>> You also seem to want the following:
>>>
>>> a. Use core.async
>>> b. Make the solution "elegant"
>>>
>>> That's all fine, but it seems like a bit of a golden hammer. Your
>>> problem is precisely not the kind of problem core.async channels are
>>> designed to solve. Channels are there to provide decoupling of producer and
>>> consumer, whereas you, in fact, need the kind of coupling provide by
>>> function calls.
>>>
>>> Channels are for when you don't want to know how or when your product is
>>> handled. You're finished, you throw the product on a channel, and you move
>>> on to making your next product. All you ever find out is if/that your
>>> product has been picked off the channel.
>>>
>>> Function calls are for when you do need to know what happened to your
>>> product, and when it has all happened, because you can't move on until the
>>> last thing has been done.
>>>
>>> Just call the middleman in synchronous code in your producer, with one
>>> value, and look at its return value to tell you what to do next. You get
>>> your blocking semantics (1-3) and you also get information about success or
>>> failure.
>>>
>>> Using channels, consumption is the only information the producer hears
>>> back. To tell the producer you've completed your mission as middleman,
>>> you'll need another channel. If you really want to use core.async:
>>>
>>> (>! c v) ; blocks until consumed
>>> (respond-to (<! ackchan)) ; blocks until handled
>>> ...do expensive stuff
>>>
>>> but this is just an expensive function call.
>>>
>>> Regards,
>>>
>>> Fergal Byrne
>>>
>>>
>>> On Thu, Oct 9, 2014 at 11:16 AM, Eduard Bondarenko <edb...@gmail.com>
>>> wrote:
>>>
>>>> I think the point is to not generate values by producer if external
>>>> service is not available
>>>> "The producer only unparks when the value is effectively consumed by
>>>> the external service. That's my objective."
>>>>
>>>> "external ready" channel here serves as a latch that stops producer
>>>> generating value.
>>>>
>>>> On Thu, Oct 9, 2014 at 6:39 AM, Fluid Dynamics <a209...@trbvm.com>
>>>> wrote:
>>>> > On Monday, October 6, 2014 9:36:59 AM UTC-4, edbond wrote:
>>>> >>
>>>> >> Add one more chan, "external ready".
>>>> >> Put :ok there to let producer generate new value.
>>>> >>
>>>> >> producer:
>>>> >> - read from "external ready"
>>>> >> - generate value
>>>> >> - put into "outgoing" chan
>>>> >>
>>>> >> client:
>>>> >> - contact external server, put in "external ready" if ok
>>>> >> - read from "outgoing" chan
>>>> >> - send to external
>>>> >>
>>>> >> Handle exceptions and loop where you need.
>>>> >
>>>> >
>>>> > If you're not reading from "outgoing" until the external server is
>>>> known to
>>>> > be ready, you don't need the "external ready" channel. Backpressure
>>>> on the
>>>> > "outgoing" channel will suffice to keep the producer from overrunning
>>>> the
>>>> > consumer in this case.
>>>> >
>>>> > --
>>>> > You received this message because you are subscribed to the Google
>>>> > Groups "Clojure" group.
>>>> > To post to this group, send email to clo...@googlegroups.com
>>>> > Note that posts from new members are moderated - please be patient
>>>> with your
>>>> > first post.
>>>> > To unsubscribe from this group, send email to
>>>> > clojure+u...@googlegroups.com
>>>> > For more options, visit this group at
>>>> > http://groups.google.com/group/clojure?hl=en
>>>> > ---
>>>> > You received this message because you are subscribed to a topic in the
>>>> > Google Groups "Clojure" group.
>>>> > To unsubscribe from this topic, visit
>>>> > https://groups.google.com/d/topic/clojure/QbiwXYDw6oA/unsubscribe.
>>>> > To unsubscribe from this group and all its topics, send an email to
>>>> > clojure+u...@googlegroups.com.
>>>> > For more options, visit https://groups.google.com/d/optout.
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Clojure" group.
>>>> To post to this group, send email to clo...@googlegroups.com
>>>> Note that posts from new members are moderated - please be patient with
>>>> your first post.
>>>> To unsubscribe from this group, send email to
>>>> clojure+u...@googlegroups.com
>>>> For more options, visit this group at
>>>> http://groups.google.com/group/clojure?hl=en
>>>> ---
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Clojure" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to clojure+u...@googlegroups.com.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> Fergal Byrne, Brenter IT
>>>
>>> http://inbits.com - Better Living through Thoughtful Technology
>>> http://ie.linkedin.com/in/fergbyrne/ - https://github.com/fergalbyrne
>>>
>>> Founder of Clortex: HTM in Clojure - https://github.com/nupic-
>>> community/clortex
>>>
>>> Author, Real Machine Intelligence with Clortex and NuPIC
>>> Read for free or buy the book at https://leanpub.com/realsmartmachines
>>>
>>> Speaking on Clortex and HTM/CLA at euroClojure Krakow, June 2014:
>>> http://euroclojure.com/2014/
>>> and at LambdaJam Chicago, July 2014: http://www.lambdajam.com
>>>
>>> e:fergalby...@gmail.com t:+353 83 4214179
>>> Join the quest for Machine Intelligence at http://numenta.org
>>> Formerly of Adnet edi...@adnet.ie http://www.adnet.ie
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Clojure" group.
>>> To post to this group, send email to clo...@googlegroups.com
>>> Note that posts from new members are moderated - please be patient with
>>> your first post.
>>> To unsubscribe from this group, send email to
>>> clojure+u...@googlegroups.com
>>> For more options, visit this group at
>>> http://groups.google.com/group/clojure?hl=en
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "Clojure" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to clojure+u...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
