Are the writes commutative? Since you are using pmap, I presume so. In that case, you could funnel the writes through an agent serving as a queue.
-Per On Thu, Mar 25, 2010 at 9:59 PM, Scott <sbuck...@gmail.com> wrote: > id prefer best practices if possible > > typically, cheating has consequences down the line > > On Mar 25, 10:43 am, Joop Kiefte <iko...@gmail.com> wrote: >> Isn't programming not all about cheating the computer in doing what you want >> it to do? In the book programming clojure you can find an example with locks >> as well. >> >> 2010/3/25 Scott <sbuck...@gmail.com> >> >> >> >> > Question on best practices on handling SQL database concurrency issues >> >> > I am "pmap"ping a evaluation to a long list (which calls a >> > computationally intense script) from within clojure. The script >> > itself is designed to be completely free of concurrency side-effects. >> > During the evaluation, several calculations are made that are then >> > written to a SQLite database. My approach is to test for concurrency >> > issues on a dual core system prior to moving to a cluster. >> >> > What I find is that on occasion there is a database locking issue when >> > sub-processes try to write to the database at the same time >> > (java.sql.SQLException: database is locked). The side effect is that >> > one of the evaluations is not written to the database (bad, cause it >> > takes 3min to compute). I can fix it by catching the exception, and >> > then calling (Thread/sleep) before trying to rewrite again. This is >> > an ugly fix, and I am concerned that this may not scale properly. >> >> > What is the best practices to handle such an issue in a concurrent and >> > scalable way? Is it as simple as moving to a better database, such as >> > mySQL? I could use Threads/Locks and move the db transaction outside >> > the evaluation loop, or save all transactions and then commit after >> > all evaluations are done. I can't help but feel both solns seem like >> > cheating when working with a conncurrent language such as clojure. >> >> > Any Advice? >> >> > I am using contrib.sql and java.sql (org.sqlite.JDBC) >> >> > -- >> > You received this message because you are subscribed to the Google >> > Groups "Clojure" group. >> > To post to this group, send email to clojure@googlegroups.com >> > Note that posts from new members are moderated - please be patient with >> > your first post. >> > To unsubscribe from this group, send email to >> > clojure+unsubscr...@googlegroups.com<clojure%2bunsubscr...@googlegroups.com> >> > For more options, visit this group at >> >http://groups.google.com/group/clojure?hl=en >> >> > To unsubscribe from this group, send email to clojure+ >> > unsubscribegooglegroups.com or reply to this email with the words "REMOVE >> > ME" as the subject. >> >> -- >> Communication is essential. So we need decent tools when communication is >> lacking, when language capability is hard to acquire... >> >> -http://esperanto.net -http://esperanto-jongeren.nl >> >> Linux-user #496644 (http://counter.li.org) - first touch of linux in 2004 > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with your > first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > > To unsubscribe from this group, send email to > clojure+unsubscribegooglegroups.com or reply to this email with the words > "REMOVE ME" as the subject. > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en To unsubscribe from this group, send email to clojure+unsubscribegooglegroups.com or reply to this email with the words "REMOVE ME" as the subject.
