Cool! I'm getting back to Clojure after an extended absence. Just today I was pondering the design of a solution to a similar problem, though I suspect our requirements diverge on several points. My tentative conclusion was that it could be done entirely in Clojure and without modifying existing code. Maybe you can poke holes in my fledging plan since you've obviously been thinking about this sort of problem longer than me:
There's a new pref reference type. It consists of a key and an atom containing nil for unloaded objects and an STM reference for loaded objects. When a pref is dereferenced, it checks its atom. If nil, it first loads the object from disk into a fresh STM reference (which has a metadata field pointing back to the pref) and mutates the atom so it points to it. In either case it finishes by dereferencing the STM reference. When a pref is mutated, it first goes through the same motions as for dereferencing. Then it simply forwards the mutation to the underlying STM reference. Watchers are installed on STM references backed by prefs. Thus we are notified when something is mutated. There is a pref-specific transaction boundary form called atomic, analogous to dosync. The watchers are used to determine which prefs were mutated during the transaction so as to flag them dirty for write-back or write-through caching; this is why we need the pref back reference in the metadata. Anyway, even assuming this all works, it will obviously be less computationally efficient than extending LockingTransaction.java with special support. -Per On Thu, Sep 23, 2010 at 10:21 AM, Alyssa Kwan <alyssa.c.k...@gmail.com> wrote: > This is in reference to > http://groups.google.com/group/clojure/browse_frm/thread/d6deac0c34d0ce28/5c1e11ec2bd52bde. > > Hi everyone! > > I have hacked the Clojure core to add durability to refs. The syntax > to create these is (dref <val> <key> <path>), where <key> and <path> > are strings. Then you use them just like refs. Creating a dref > creates a global identity, such that subsequent dref calls to the same > key and path will get the same dref. On subsequent dref calls, the > <val> will be ignored and the persisted value used. This includes in > subsequent VM instances. > > Get it here: git://github.com/kwanalyssa/clojure.git > > 1. <path> refers to BDB JE databases which get created in the "data" > directory of your project. > 2. BDB JE is used. I'm ignorant of IP and licensing issues. Making > BDB JE core to Clojure is probably an issue. > 3. Currently only a subset of Clojure primitives types is supported. > No BigDecimal or Ratio yet. See the comprehensive list (and > serialization mappings) at the bottom of src/jvm/clojure/lang/ > DRef.java. BDB JE TupleBindings are used. Submissions welcome, > especially for the persistent data structures. > 4. How do we approach the problem of storing objects with lexical > environments? > 5. Unit tests welcome! I didn't do TDD since the work is in Java and > there's no Java-level tests in the project. Please add your own in > test/clojure/test_clojure/drefs.clj. I'm new to concurrency, so tests > along those lines would be awesome! > 6. The ACID part is not really guaranteed!!! STM is currently one- > phase-commit. I inserted two-phase-commits to the data stores in the > middle of the one-phase-commit. There's the remote possibility that > STM in-memory changes fail AFTER writing to disk. It's REALLY remote, > but it is possible. STM would have to be made 2PC to make this > airtight. That's way beyond my current grasp of both concurrency and > Clojure implementation. > 7. I was aiming for an API where <path> is optional. However, I > didn't want to stray from the ref API, which has variable arity. > Suggestions on how to reconcile the two are welcome! > 8. To maintain global identity, I use a static cache, which requires > non-hard references to avoid OOM issues. This is my first time doing > this, so please check my code to make sure that I'm doing it right. > I'm using SoftReferences, though WeakReferences may be better for real- > life usage patterns. Let me know! > > Please dig in! Feedback appreciated! > Alyssa Kwan > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with your > first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
