Hi Brian, On Wednesday, 23 October 2013 05:33:30 UTC+5:30, Brian Craft wrote: > > I'm doing a load to db, which looks roughly like "read, transform, insert, > repeat". > > Blocking on the inserts leaves the cores sitting cold while they could be > doing the next read/transform. It's tempting to try an agent for the > inserts. If I understand them correctly, that would queue the inserts as > they are ready, allowing the read/transform to continue. I'm concerned that > this would just fill memory, as the inserts can't keep up with the reads. >
Since the inserts are for the same kind of data, one idea could be to use ArrayBlockingQueue or LinkedBlockingQueue to queue the inserts and another thread (possibly a future and optionally an agent) could retrieve this data to insert to the database. Batching the inserts (with right batch size) may be faster than individual inserts. The thread that reads from the DB can track the level of the insert queue and pause to avoid over-feeding. Shantanu -- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
