On Thursday, October 9, 2014 10:25:38 PM UTC+2, Nahuel Greco wrote:
>
> Leon: your :useless example is equivalent to one using the hypothetical
> peek operation and to the ack-channel solution, but what you define as
> ":useless multithreading" because at one time only one thread is working,
> is a valid synchronization scenario and the one I was searching for. The
> objective here was not to max CPU usage with these two threads / go-blocks.
>
What I wanted to illustrate with useless multithreading was my earlier
point that for your requirements I can't find a reason for multithreading
and hence no reason for synchronization.
> As I said in a previous mail "maybe you want to reserve the cpu cycles
> for other tasks, or the producer needs a confirmation for other purposes
> before computing or requesting the value from another party, this is a
> distilled example".
>
What difference should this make? I have already integrated these
requirements in my mail from Oct. 5. You can put the confirmation part
between steps 3 and 4.
> Also think the producer after sending the computed value and before
> sending the :useless signal can do other things, maybe contacting other
> services to pre-fetch data used in the next value.
>
It could, because it is unblocked.
But i f you had used peek!, it couldn't. It would just have to pre-fetch
its data and peek! would be blocked until its done and has produced a
value. (I hope the discussion is still about finding a valid reason why
peek! would be a useful operation in core.async)
>
> Also the :useless example needs more code on the producer side than a
> simple (>! c v). Given there is no peek operation and the producer code
> must participate explicitly in the synchronization, I prefer the ack
> solution over the :useless one.
>
The producer must participate in the synchronization in the ack solution as
well. There is no peek operation in the ack solution as well.
You could as well let the producer put the value twice and every first take
would be your peak.
I don't recommend :useless or an ack channel as a "solution" over the
other. My point is that if this problem was real, using multiple threads as
a solution and the resulting effort to coordinate them so that only one of
them would run at a time doesn't seem to provide benefits in terms of
performance, abstraction or readability.
Kind regards,
Leon.
You are right about impossibility of using alts! in the producer code to
check if the chan can be written to before computing the value. I forgot
alts! is a function, not a macro, and you can't pass expressions to it to
be evaluated when the chan becomes available to write.
Your last sample seems an useful way to manage simple stateless
single-value producers from the consumer, but producers can be long-running
go-blocks with their own state and with channels to other services, so I
think is a little off base the original question.
>
>
> Saludos,
> Nahuel Greco.
>
> On Thu, Oct 9, 2014 at 4:16 PM, Leon Grapenthin <grapent...@gmail.com
> <javascript:>> wrote:
>
>> On Thursday, October 9, 2014 2:22:50 PM UTC+2, Nahuel Greco wrote:
>>>
>>> Fluid: as you said, backpressure on the outgoing channel will be
>>> sufficient to avoid producer to overrun consumer (without using any extra
>>> ack-channel), but the producer will compute a new not-yet-used value before
>>> blocking on sending it to consumer. That's what I want to avoid.
>>>
>>
>> You can avoid this very easily. Everytime before the producer expensively
>> calculates its next value to put it on the channel, make it put a cheaply
>> calculated value, e. g. the keyword :useless, on the channel instead.
>>
>> Now on the consumer side you can do this:
>>
>> 0. Blocking (immediate take): Consume the keyword :useless = The producer
>> will begin production
>> 1. *This is what you want from peek! *Blocking take = Waiting for the
>> producer to have produced the expensive value, consume it *but the
>> producer won't calculate the next value (because he puts the :useless
>> keyword which won't be consumed until later)*
>> 2. Wait for external service to be ready, put value there
>> 3. Go to step 1 trigger production of the next value
>>
>> Assuming the producer and the consumer run on different threads, what
>> have you won by introducing step 0?
>>
>> 1. The producer thread can't begin to work until the consumer thread did
>> his work
>> 2. The consumer thread can't begin to work until the producer thread did
>> his work
>>
>> = At every point in time only one thread performs. Entirely :useless
>> multi-threading.
>>
>> I can also use alts! on the producer to check if he can write to the
>>> channel before computing the value, [...]
>>>
>> You can't
>>
>>
>>
>>> [...]
>>>
>>
>>
>>> Fregal: If middleman is managing all the communication with the external
>>> service for *multiple* producers, then it can't be reduced to a simple
>>> synchronous function call from a producer, hence the necessity of
>>> core.async (the alternative is to descend to locks world).
>>>
>>
>> If I understand you right, you would want to start production of multiple
>> values in parallel, wait for the first to be ready, then put that to the
>> external service and loop. Here core.async can be of help. E. g.
>> (defn exp-calcs
>> []
>> #{(a/thread calc-exp-val-1)
>> (a/thread calc-exp-val-2)
>> (a/thread calc-exp-val-3)
>> (a/thread calc-exp-val-4)
>> (a/thread calc-exp-val-5)
>> ; ...
>> })
>>
>> (loop [chans (exp-calcs)]
>> (let [[val ch] (a/alts!! (seq exp-calcs))]
>> ;; wait for external service, put val there
>>
>> ;; also have the cancellation ch in chans and use it to guard the
>> ;; following form
>> (let [chans (disj chans ch)]
>> (recur (if (seq chans)
>> chans
>> (exp-calcs))))))
>>
>>
>>
>> Again, no peek required.
>>
>>
>>>
>>>
>>>
>>> Saludos,
>>> Nahuel Greco.
>>>
>>> On Thu, Oct 9, 2014 at 9:07 AM, Fergal Byrne <fergalby...@gmail.com>
>>> wrote:
>>>
>>>> Hi Nahuel,
>>>>
>>>> I think it's worth stepping back from discussing solutions and have a
>>>> look at the problem. If I'm reading things right you need the following:
>>>>
>>>> 1. Producer produces values one at a time - should not step until last
>>>> value has been handled correctly elsewhere.
>>>> 2. Middleman needs to consume one value and (potentially) retry an
>>>> external service until it can verify correct handling.
>>>> 3. Producer should not interpret the consumption of a value as a signal
>>>> to proceed.
>>>>
>>>> You also seem to want the following:
>>>>
>>>> a. Use core.async
>>>> b. Make the solution "elegant"
>>>>
>>>> That's all fine, but it seems like a bit of a golden hammer. Your
>>>> problem is precisely not the kind of problem core.async channels are
>>>> designed to solve. Channels are there to provide decoupling of producer
>>>> and
>>>> consumer, whereas you, in fact, need the kind of coupling provide by
>>>> function calls.
>>>>
>>>> Channels are for when you don't want to know how or when your product
>>>> is handled. You're finished, you throw the product on a channel, and you
>>>> move on to making your next product. All you ever find out is if/that your
>>>> product has been picked off the channel.
>>>>
>>>> Function calls are for when you do need to know what happened to your
>>>> product, and when it has all happened, because you can't move on until the
>>>> last thing has been done.
>>>>
>>>> Just call the middleman in synchronous code in your producer, with one
>>>> value, and look at its return value to tell you what to do next. You get
>>>> your blocking semantics (1-3) and you also get information about success
>>>> or
>>>> failure.
>>>>
>>>> Using channels, consumption is the only information the producer hears
>>>> back. To tell the producer you've completed your mission as middleman,
>>>> you'll need another channel. If you really want to use core.async:
>>>>
>>>> (>! c v) ; blocks until consumed
>>>> (respond-to (<! ackchan)) ; blocks until handled
>>>> ...do expensive stuff
>>>>
>>>> but this is just an expensive function call.
>>>>
>>>> Regards,
>>>>
>>>> Fergal Byrne
>>>>
>>>>
>>>> On Thu, Oct 9, 2014 at 11:16 AM, Eduard Bondarenko <edb...@gmail.com>
>>>> wrote:
>>>>
>>>>> I think the point is to not generate values by producer if external
>>>>> service is not available
>>>>> "The producer only unparks when the value is effectively consumed by
>>>>> the external service. That's my objective."
>>>>>
>>>>> "external ready" channel here serves as a latch that stops producer
>>>>> generating value.
>>>>>
>>>>> On Thu, Oct 9, 2014 at 6:39 AM, Fluid Dynamics <a209...@trbvm.com>
>>>>> wrote:
>>>>> > On Monday, October 6, 2014 9:36:59 AM UTC-4, edbond wrote:
>>>>> >>
>>>>> >> Add one more chan, "external ready".
>>>>> >> Put :ok there to let producer generate new value.
>>>>> >>
>>>>> >> producer:
>>>>> >> - read from "external ready"
>>>>> >> - generate value
>>>>> >> - put into "outgoing" chan
>>>>> >>
>>>>> >> client:
>>>>> >> - contact external server, put in "external ready" if ok
>>>>> >> - read from "outgoing" chan
>>>>> >> - send to external
>>>>> >>
>>>>> >> Handle exceptions and loop where you need.
>>>>> >
>>>>> >
>>>>> > If you're not reading from "outgoing" until the external server is
>>>>> known to
>>>>> > be ready, you don't need the "external ready" channel. Backpressure
>>>>> on the
>>>>> > "outgoing" channel will suffice to keep the producer from
>>>>> overrunning the
>>>>> > consumer in this case.
>>>>> >
>>>>> > --
>>>>> > You received this message because you are subscribed to the Google
>>>>> > Groups "Clojure" group.
>>>>> > To post to this group, send email to clo...@googlegroups.com
>>>>> > Note that posts from new members are moderated - please be patient
>>>>> with your
>>>>> > first post.
>>>>> > To unsubscribe from this group, send email to
>>>>> > clojure+u...@googlegroups.com
>>>>> > For more options, visit this group at
>>>>> > http://groups.google.com/group/clojure?hl=en
>>>>> > ---
>>>>> > You received this message because you are subscribed to a topic in
>>>>> the
>>>>> > Google Groups "Clojure" group.
>>>>> > To unsubscribe from this topic, visit
>>>>> > https://groups.google.com/d/topic/clojure/QbiwXYDw6oA/unsubscribe.
>>>>> > To unsubscribe from this group and all its topics, send an email to
>>>>> > clojure+u...@googlegroups.com.
>>>>> > For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "Clojure" group.
>>>>> To post to this group, send email to clo...@googlegroups.com
>>>>> Note that posts from new members are moderated - please be patient
>>>>> with your first post.
>>>>> To unsubscribe from this group, send email to
>>>>> clojure+u...@googlegroups.com
>>>>> For more options, visit this group at
>>>>> http://groups.google.com/group/clojure?hl=en
>>>>> ---
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "Clojure" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to clojure+u...@googlegroups.com.
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Fergal Byrne, Brenter IT
>>>>
>>>> http://inbits.com - Better Living through Thoughtful Technology
>>>> http://ie.linkedin.com/in/fergbyrne/ - https://github.com/fergalbyrne
>>>>
>>>> Founder of Clortex: HTM in Clojure - https://github.com/nupic-
>>>> community/clortex
>>>>
>>>> Author, Real Machine Intelligence with Clortex and NuPIC
>>>> Read for free or buy the book at https://leanpub.com/realsmartmachines
>>>>
>>>> Speaking on Clortex and HTM/CLA at euroClojure Krakow, June 2014:
>>>> http://euroclojure.com/2014/
>>>> and at LambdaJam Chicago, July 2014: http://www.lambdajam.com
>>>>
>>>> e:fergalby...@gmail.com t:+353 83 4214179
>>>> Join the quest for Machine Intelligence at http://numenta.org
>>>> Formerly of Adnet edi...@adnet.ie http://www.adnet.ie
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Clojure" group.
>>>> To post to this group, send email to clo...@googlegroups.com
>>>> Note that posts from new members are moderated - please be patient with
>>>> your first post.
>>>> To unsubscribe from this group, send email to
>>>> clojure+u...@googlegroups.com
>>>> For more options, visit this group at
>>>> http://groups.google.com/group/clojure?hl=en
>>>> ---
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Clojure" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to clojure+u...@googlegroups.com.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clo...@googlegroups.com
>> <javascript:>
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+u...@googlegroups.com <javascript:>
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to clojure+u...@googlegroups.com <javascript:>.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
