Sorry. I put close at he and of the main method but I afraid the main
thread will reach close method before all the data will be put to the queue.
Is any way to check a state of the queue? If I handle channels returned by
all *-sub-methods with go inside I could check state if was finished?
Regards,
J
On 6 December 2017 at 12:21, Jacek Grzebyta <grzebyta....@gmail.com> wrote:
> Thanks a lot.
>
> I found you changed the finish slightly - you close q within go body.
> Where should I put close! in case if I load a channel in many places within
> several methods? I afraid the go body is done
>
> E.g.:
>
> (defn main-method [db-api]
> (let [main-queue (chan 20000) ]
> (sink-method main-queue db-api) ;; population db within go-loop body
> (loading-method-1 main-queue & rest)
> (loading-method-n main-queue & rest)))
>
> And methods pattern is like:
>
> (def loading-method-.. [ q ....]
> (doseq
> (loading-sub-method q)
> (map
> #(doseq
>
> (go
> >! q (processing i j k l m))
> ) .. ) ... ))
>
>
> On 6 December 2017 at 11:50, Ray Miller <r...@1729.org.uk> wrote:
>
>> On 6 December 2017 at 11:23, Jacek Grzebyta <grzebyta....@gmail.com>
>> wrote:
>>>
>>> I have to populate a triple store with a big number of data (~38k
>>> records x 12) and there is a deadly narrow bottleneck - IO operations
>>> speed. To fix it I did:
>>> 1. To avoid threads overflow I put all compute results into channel.
>>> 2. Loading data in chunks is better than single transaction for
>>> single record
>>>
>>> I tried to do by creating channel with poputale-all traversal but it
>>> seems doesn't work properly. In the following mock example it works when
>>> the chunk size is equal the data vector (i.e. 6): "value: [of made is
>>> fruit soup Berry]" - for now I do not care the order.
>>>
>>> (let [q (a/chan 500 (partition-all 6))
>>> in ["Berry" "soup" "is" "made" "of" "fruit"]]
>>> (a/go-loop [j (a/<! q)]
>>> (when j
>>> (println "value: " j)
>>> (recur (a/<! q))))
>>> (doseq [itm in]
>>> (a/go (a/>! q itm))))
>>>
>>>
>>> I cannot see any problem. How can I solve it? In the following example
>>> chunk size should be max 6? I expected partition-all will work the same way
>>> as itself:
>>>
>>> (partition-all 5 ["Berry" "soup" "is" "made" "of" "fruit"]) ==>
>>> (("Berry" "soup" "is" "made" "of") ("fruit"))
>>>
>>>
>> I think you just need to close the channel when you've finished
>> populating it:
>>
>> (let [q (a/chan 500 (partition-all 5))
>> in ["Berry" "soup" "is" "made" "of" "fruit"]]
>> (a/go-loop [j (a/<! q)]
>> (when j
>> (println "value: " j)
>> (recur (a/<! q))))
>> (a/go
>> (doseq [itm in]
>> (a/>! q itm))
>> (a/close! q)))
>>
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clojure@googlegroups.com
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+unsubscr...@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to clojure+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
