Hi Nahuel, If you look at the definition of "simple" it means "not compound". In your case, you are trying to have a single thing (the channel) do two jobs: convey the value, and control the producer. This is not simple, and makes the producer trust that the middleman will always respect this hidden dependence. It also hides your intent elsewhere (anyone reading your code would assume that the producer does not have to wait).
In this case (the ack channel), you are adding a couple of lines of code (to make the ack channel and to block until it comes back), but you are simplifying the producer when you do this, not complicating it. The (!< c [v ack]) does one thing - I'm finished my expensive op, please deal with it. The (respond-to (<! ack)) does one thing - wait until the value has been processed, now I can move on. The code is self-documenting. The producer finishes its work, puts the stuff on a channel with a self-addressed envelope, and then immediately blocks for the ack. It couldn't possibly be simpler than this. The decoupling channels provide is exactly that: everything except backpressure is decoupled. You're trying to recouple the producer and middleman, by trying to make a non-atomic thing atomic. You're also designing the middleman based on the needs of the producer. With your design, the middleman can only handle one product at a time. With the put-ack design, the middleman could have a longer queue, construct batches of similar values, and more efficiently use the external service. You're also creating a bottleneck, because other producers are all blocked until your middleman has got the external service to work with your value. Either the external service's ability to complete is based on something in your value or not. If not, then have the middleman wait until the service is ready (as someone suggested) before taking from the channel - this achieves your goals with backpressure. If you do need to know something in the value, then take the value and spin up a go block to retry the service (passing it the ack channel, so it can unblock the producer), and move on. Adding peek to channels couldn't be hard (it's open source after all!), so you could do it for yourself if you think it would give you any value. But I can give you a fairly precise estimate on how long a PR for this would sit in Rich's in-tray.. Cheers, Fergal On Thu, Oct 9, 2014 at 2:01 PM, Nahuel Greco <ngr...@gmail.com> wrote: > Fergal: I knew about the ack channel solution (note I mentioned it a while > couple of mails ago), I was trying to avoid it with the objective of making > the producer code extra simple. > > Also note, in core.async the producers aren't totally decoupled from > consumers, there is an implicit consumer->producer signalling in the form > of channel backpressure. The only way to decouple them totally is to use > "infinitely" sized or dropping queues (something like Erlang tries to do > given enough memory, where the only form of backpressure are explicit > acks). The peek operation I proposed is a way to fine control that > backpressure signalling. > > In POSIX the select() operation doesn't peeks the value but can be used to > know if there is a value present without consuming it. In contrast, I think > alts! complects three operations: 1- blocking until new value / can write > one 2- reading/putting it 3- when reading, consuming it. > > Besides the alternative solutions for the producer code (waiting on ack > channel, alts! before computing the value) my original question remains > unanswered: There is something in core.async implementation/design/roadmap > preventing adding a peek operation? I was hoping to avoid a deep dive on > core.async code :) > > > > Saludos, > Nahuel Greco. > > On Thu, Oct 9, 2014 at 9:39 AM, Fergal Byrne <fergalbyrnedub...@gmail.com> > wrote: > >> Hi Nahuel, >> >> Thanks for the clarification. Multiple producers, single middleman is a >> different problem from the one we (or at least I) thought we were dealing >> with. In that case, the put your ack channel in your product, block on the >> ack channel method is the right one. >> >> (loop [...] >> ..something expensive... >> (let [ack (chan)] >> (>! c [v ack]) >> (respond-to (<! ack)) >> (recur ...)) >> >> Regards, >> >> Fergal >> >> On Thu, Oct 9, 2014 at 1:21 PM, Nahuel Greco <ngr...@gmail.com> wrote: >> >>> Fluid: as you said, backpressure on the outgoing channel will be >>> sufficient to avoid producer to overrun consumer (without using any extra >>> ack-channel), but the producer will compute a new not-yet-used value before >>> blocking on sending it to consumer. That's what I want to avoid. >>> >>> I can also use alts! on the producer to check if he can write to the >>> channel before computing the value, but I wanted to keep the code in the >>> producer in the simplest form possible (a simple (>! c v) parking until all >>> parties consume the value, avoiding an ack-chan or an alts!) . >>> >>> Eduard: Exactly. >>> >>> Fregal: If middleman is managing all the communication with the external >>> service for *multiple* producers, then it can't be reduced to a simple >>> synchronous function call from a producer, hence the necessity of >>> core.async (the alternative is to descend to locks world). >>> >>> >>> >>> Saludos, >>> Nahuel Greco. >>> >>> On Thu, Oct 9, 2014 at 9:07 AM, Fergal Byrne < >>> fergalbyrnedub...@gmail.com> wrote: >>> >>>> Hi Nahuel, >>>> >>>> I think it's worth stepping back from discussing solutions and have a >>>> look at the problem. If I'm reading things right you need the following: >>>> >>>> 1. Producer produces values one at a time - should not step until last >>>> value has been handled correctly elsewhere. >>>> 2. Middleman needs to consume one value and (potentially) retry an >>>> external service until it can verify correct handling. >>>> 3. Producer should not interpret the consumption of a value as a signal >>>> to proceed. >>>> >>>> You also seem to want the following: >>>> >>>> a. Use core.async >>>> b. Make the solution "elegant" >>>> >>>> That's all fine, but it seems like a bit of a golden hammer. Your >>>> problem is precisely not the kind of problem core.async channels are >>>> designed to solve. Channels are there to provide decoupling of producer and >>>> consumer, whereas you, in fact, need the kind of coupling provide by >>>> function calls. >>>> >>>> Channels are for when you don't want to know how or when your product >>>> is handled. You're finished, you throw the product on a channel, and you >>>> move on to making your next product. All you ever find out is if/that your >>>> product has been picked off the channel. >>>> >>>> Function calls are for when you do need to know what happened to your >>>> product, and when it has all happened, because you can't move on until the >>>> last thing has been done. >>>> >>>> Just call the middleman in synchronous code in your producer, with one >>>> value, and look at its return value to tell you what to do next. You get >>>> your blocking semantics (1-3) and you also get information about success or >>>> failure. >>>> >>>> Using channels, consumption is the only information the producer hears >>>> back. To tell the producer you've completed your mission as middleman, >>>> you'll need another channel. If you really want to use core.async: >>>> >>>> (>! c v) ; blocks until consumed >>>> (respond-to (<! ackchan)) ; blocks until handled >>>> ...do expensive stuff >>>> >>>> but this is just an expensive function call. >>>> >>>> Regards, >>>> >>>> Fergal Byrne >>>> >>>> >>>> On Thu, Oct 9, 2014 at 11:16 AM, Eduard Bondarenko <edb...@gmail.com> >>>> wrote: >>>> >>>>> I think the point is to not generate values by producer if external >>>>> service is not available >>>>> "The producer only unparks when the value is effectively consumed by >>>>> the external service. That's my objective." >>>>> >>>>> "external ready" channel here serves as a latch that stops producer >>>>> generating value. >>>>> >>>>> On Thu, Oct 9, 2014 at 6:39 AM, Fluid Dynamics <a2093...@trbvm.com> >>>>> wrote: >>>>> > On Monday, October 6, 2014 9:36:59 AM UTC-4, edbond wrote: >>>>> >> >>>>> >> Add one more chan, "external ready". >>>>> >> Put :ok there to let producer generate new value. >>>>> >> >>>>> >> producer: >>>>> >> - read from "external ready" >>>>> >> - generate value >>>>> >> - put into "outgoing" chan >>>>> >> >>>>> >> client: >>>>> >> - contact external server, put in "external ready" if ok >>>>> >> - read from "outgoing" chan >>>>> >> - send to external >>>>> >> >>>>> >> Handle exceptions and loop where you need. >>>>> > >>>>> > >>>>> > If you're not reading from "outgoing" until the external server is >>>>> known to >>>>> > be ready, you don't need the "external ready" channel. Backpressure >>>>> on the >>>>> > "outgoing" channel will suffice to keep the producer from >>>>> overrunning the >>>>> > consumer in this case. >>>>> > >>>>> > -- >>>>> > You received this message because you are subscribed to the Google >>>>> > Groups "Clojure" group. >>>>> > To post to this group, send email to clojure@googlegroups.com >>>>> > Note that posts from new members are moderated - please be patient >>>>> with your >>>>> > first post. >>>>> > To unsubscribe from this group, send email to >>>>> > clojure+unsubscr...@googlegroups.com >>>>> > For more options, visit this group at >>>>> > http://groups.google.com/group/clojure?hl=en >>>>> > --- >>>>> > You received this message because you are subscribed to a topic in >>>>> the >>>>> > Google Groups "Clojure" group. >>>>> > To unsubscribe from this topic, visit >>>>> > https://groups.google.com/d/topic/clojure/QbiwXYDw6oA/unsubscribe. >>>>> > To unsubscribe from this group and all its topics, send an email to >>>>> > clojure+unsubscr...@googlegroups.com. >>>>> > For more options, visit https://groups.google.com/d/optout. >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Clojure" group. >>>>> To post to this group, send email to clojure@googlegroups.com >>>>> Note that posts from new members are moderated - please be patient >>>>> with your first post. >>>>> To unsubscribe from this group, send email to >>>>> clojure+unsubscr...@googlegroups.com >>>>> For more options, visit this group at >>>>> http://groups.google.com/group/clojure?hl=en >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Clojure" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to clojure+unsubscr...@googlegroups.com. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> >>>> >>>> -- >>>> >>>> Fergal Byrne, Brenter IT >>>> >>>> http://inbits.com - Better Living through Thoughtful Technology >>>> http://ie.linkedin.com/in/fergbyrne/ - https://github.com/fergalbyrne >>>> >>>> Founder of Clortex: HTM in Clojure - >>>> https://github.com/nupic-community/clortex >>>> >>>> Author, Real Machine Intelligence with Clortex and NuPIC >>>> Read for free or buy the book at https://leanpub.com/realsmartmachines >>>> >>>> Speaking on Clortex and HTM/CLA at euroClojure Krakow, June 2014: >>>> http://euroclojure.com/2014/ >>>> and at LambdaJam Chicago, July 2014: http://www.lambdajam.com >>>> >>>> e:fergalbyrnedub...@gmail.com t:+353 83 4214179 >>>> Join the quest for Machine Intelligence at http://numenta.org >>>> Formerly of Adnet edi...@adnet.ie http://www.adnet.ie >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Clojure" group. >>>> To post to this group, send email to clojure@googlegroups.com >>>> Note that posts from new members are moderated - please be patient with >>>> your first post. >>>> To unsubscribe from this group, send email to >>>> clojure+unsubscr...@googlegroups.com >>>> For more options, visit this group at >>>> http://groups.google.com/group/clojure?hl=en >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "Clojure" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to clojure+unsubscr...@googlegroups.com. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Clojure" group. >>> To post to this group, send email to clojure@googlegroups.com >>> Note that posts from new members are moderated - please be patient with >>> your first post. >>> To unsubscribe from this group, send email to >>> clojure+unsubscr...@googlegroups.com >>> For more options, visit this group at >>> http://groups.google.com/group/clojure?hl=en >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "Clojure" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to clojure+unsubscr...@googlegroups.com. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> >> -- >> >> Fergal Byrne, Brenter IT >> >> http://inbits.com - Better Living through Thoughtful Technology >> http://ie.linkedin.com/in/fergbyrne/ - https://github.com/fergalbyrne >> >> Founder of Clortex: HTM in Clojure - >> https://github.com/nupic-community/clortex >> >> Author, Real Machine Intelligence with Clortex and NuPIC >> Read for free or buy the book at https://leanpub.com/realsmartmachines >> >> Speaking on Clortex and HTM/CLA at euroClojure Krakow, June 2014: >> http://euroclojure.com/2014/ >> and at LambdaJam Chicago, July 2014: http://www.lambdajam.com >> >> e:fergalbyrnedub...@gmail.com t:+353 83 4214179 >> Join the quest for Machine Intelligence at http://numenta.org >> Formerly of Adnet edi...@adnet.ie http://www.adnet.ie >> >> -- >> You received this message because you are subscribed to the Google >> Groups "Clojure" group. >> To post to this group, send email to clojure@googlegroups.com >> Note that posts from new members are moderated - please be patient with >> your first post. >> To unsubscribe from this group, send email to >> clojure+unsubscr...@googlegroups.com >> For more options, visit this group at >> http://groups.google.com/group/clojure?hl=en >> --- >> You received this message because you are subscribed to the Google Groups >> "Clojure" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to clojure+unsubscr...@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- Fergal Byrne, Brenter IT http://inbits.com - Better Living through Thoughtful Technology http://ie.linkedin.com/in/fergbyrne/ - https://github.com/fergalbyrne Founder of Clortex: HTM in Clojure - https://github.com/nupic-community/clortex Author, Real Machine Intelligence with Clortex and NuPIC Read for free or buy the book at https://leanpub.com/realsmartmachines Speaking on Clortex and HTM/CLA at euroClojure Krakow, June 2014: http://euroclojure.com/2014/ and at LambdaJam Chicago, July 2014: http://www.lambdajam.com e:fergalbyrnedub...@gmail.com t:+353 83 4214179 Join the quest for Machine Intelligence at http://numenta.org Formerly of Adnet edi...@adnet.ie http://www.adnet.ie -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
