Re: core.async consumer + producer working by chunk?

2018-01-06 Thread Brian J. Rubinton
https://dev.clojure.org/jira/browse/ASYNC-210 <https://dev.clojure.org/jira/browse/ASYNC-210> > On Jan 6, 2018, at 12:11 PM, Brian J. Rubinton > wrote: > > Thanks! I will. Just signed the CA. > > > On Sat, Jan 6, 2018, 12:10 PM Alex Miller <mailto:a...@pure

Re: core.async consumer + producer working by chunk?

2018-01-06 Thread Brian J. Rubinton
Thanks! I will. Just signed the CA. On Sat, Jan 6, 2018, 12:10 PM Alex Miller wrote: > > > On Saturday, January 6, 2018 at 10:56:06 AM UTC-6, Brian J. Rubinton wrote: >> >> Alex - it makes sense to me that the buffer temporarily expands beyond >> its normal s

Re: core.async consumer + producer working by chunk?

2018-01-06 Thread Brian J. Rubinton
Typo — I meant to say the channel executes puts during a take! even though the buffer is full before executing the puts. This is clearer in code (please see the gist). > On Jan 6, 2018, at 11:55 AM, Brian J. Rubinton > wrote: > > Alex - it makes sense to me that the buffer

Re: core.async consumer + producer working by chunk?

2018-01-06 Thread Brian J. Rubinton
annel? > > I will do that. My current producer is doing too many other things, but if I > break it up into separate threads or go blocks for each work queue, then that > should work. Thank you. > > > On Saturday, January 6, 2018 at 8:22:34 AM UTC-5, Brian J. Rubinton wrote

Re: core.async consumer + producer working by chunk?

2018-01-06 Thread Brian J. Rubinton
Rob - I’d go with Gary's approach, which essentially moves the splitting up of the chunk of results from the core.async channel’s transducer to the producing function. You can do that using a channel with a fixed buffer of 50 and >!!. As long as the next db query is blocked until each of the res

Re: core.async consumer + producer working by chunk?

2018-01-05 Thread Brian J. Rubinton
I don’t know; I don’t fully understand the implementation differences of >!! and offer!. The behavior of offer! makes me think the buffer is not empty until all the outputs of the transducer are consumed, but the behavior of >!! makes me think otherwise. Moritz - is the buffer cleared if: - it’

Re: core.async consumer + producer working by chunk?

2018-01-05 Thread Brian J. Rubinton
; ( ( (offer! work-queue (range 50)) nil user> (dotimes [_ 48] ( (offer! work-queue (range 50)) true user> ( On Jan 5, 2018, at 2:10 PM, Rob Nikander wrote: > > > > On Friday, January 5, 2018 at 2:03:00 PM UTC-5, Brian J. Rubinton wrote: > > What is the buffered channel’s

Re: core.async consumer + producer working by chunk?

2018-01-05 Thread Brian J. Rubinton
Hi Rob, What is the buffered channel’s buffer used for? If that’s set to 1 and the channel’s transducer is `(mapcat identity)` then the producer should be able to continuously put chunks of work onto the channel with the puts only completing when the previous chunk is completely consumed. That