Alex, yeah that explains it for me. I'll probably want to use a fully connected address space, via a mesh or a hub and spoke topology, that I fully manage - so this is the perfect level of abstraction for me. I'd like to see that ZQ implementation though.
So, to be clear, if node A sends schism-map M to node B, both A and B update M, B sends M back to A, A converges M1 and M2: If M1 and M2 have destructive conflicts, does the most "recent" change win? Or does the local copy win? Thanks, V/r John John On Thu, Apr 19, 2018 at 8:55 AM, Alex Redington <areding...@gmail.com> wrote: > I'll try to answer this question and John's at the same time. > > Schism does not try to manage state for you over time; there are a lot of > good tools for doing that already (atoms, refs, agents, channels, etc). > Schism is a set of collections that augment basic clojure collections with > enough additional information that you can take two persistent collections > with common ancestry and merge them together. So, you could take a schism > map held in an atom, dereference it and send it from a pedestal process to > an om application as edn, and read-string it in the om app to have a > replicated copy of the map. The om app could make some changes (dissoc one > key, update another) without communicating to the pedestal app for each > operation. Then send it back to the pedestal app in a POST body, and the > pedestal app would swap! the atom using schism.core/converge and the value > it just received from om. The changes, both additive and destructive, would > be replicated in the atom's value. > > So, yes, you could build a serverless chatroom where each client held an > atom with a schism collection, and they sent their copies around using > WebRTC to stay in sync. You could push a schism edn serialization over a > message queue to be synchronized on the other end. I anticipate pushing > each discrete state over the mq with an add-watch hook might yield bad > performance for little semantic gain. Schism collections are necessarily > more expensive than Clojure collections in serialization size, and > synchronization is where all of the signficant computational expense of > these kinds of data structures resides, so doing it less frequently than > every discrete update will probably be best. (A debounced send & sync that > guaranteed transmission after n milliseconds of being inert would make > sense to me) > > I'm not presently confronting a problem that these solve in my day job; my > intent was to build and have the tool ready if and when we wanted to have > better answers for "offline sync" in a single page ClojureScript > application. > > Thank you for the great questions! > > -Alex > > On Wednesday, April 18, 2018 at 9:55:36 PM UTC-4, Luke Burton wrote: >> >> >> This is cool! >> >> There seems to be some room for a recommendation on how to apply these >> data structures effectively in practice, especially around the distribution >> mechanism. For instance, I can imagine storing these in an atom and using >> watchers to push changes out over zero-mq. Or is that the Worst Idea Ever™? >> I feel like the transit layer probably has *some* impact on whether CRDTs >> are the right solution to the problem, and I'm interested to know what >> those boundaries are. >> >> Have you used them in production over the network, and if so, what >> solution did you use? >> >> Thanks again! >> >> >> >> On Apr 18, 2018, at 5:57 PM, Alex Redington <aredi...@gmail.com> wrote: >> >> Good evening! >> >> I submit for your evaluation and reasoned feedback a library I've been >> working on to provide a set of convergent replicated data types to Clojure >> and ClojureScript: >> >> https://github.com/aredington/schism >> [com.holychao/schism "0.1.0"] >> >> Schism provides convergent collections for sets, vectors, lists, and >> maps, along with edn readers and writers out of the gate. My intent is to >> provide you with all the tools necessary to replicate a collection across >> two or more computing nodes. >> >> Replication is a hard problem, so there are a few caveats to these tools >> as I provide them: >> >> - You must identify nodes within your cluster. You may choose to identify >> nodes with a random UUID, in which case schism will help you to do so. Node >> identifiers must be clojure serializable data. >> - Schism purposefully avoids carrying around a monotonically increasing >> historical collection of data about deleted entries. Consequently there are >> some ambiguities during convergence that may not exactly mirror local >> modification. >> - Schism is only solving the problem of synchronizing two in memory sets. >> Maintaining identity of those two sets, tracking state changes, and long >> term durability are responsibilities left to the schism user. >> >> Schism collections are persistent collections, so you should feel free to >> work with them as a drop in replacement for a function which would work >> against a Clojure collection. The usual utilities such as conj, assoc, >> dissoc, disj, and rest are pure functions which will return you derived >> copies, implicitly soing the convergence bookkeeping necessary in the >> background. As you work with it, schism will maintain node and timestamp >> information, with the goal of convergence providing the same result as if >> all previous invocations had occurred on one local collection in memory. >> Schism's requirements are your expectations of a Clojure collection, so >> hash, =, and support for meta are all included, as well as many other >> functions defined against Clojure's own collections. Particularly with hash >> and =, you can expect a schism collection to return the same hashcode as >> its Clojure equivalent, and to determine equality in the same way as its >> Clojure equivalent would. >> >> I've built schism in the absence of a direct itch to scratch with it. >> Should you find that it betrays your expectations of Clojure collections, >> I'd greatly appreciate a bug report and will work to correct schism quickly. >> >> I hope that it assists you in solving your own problems. >> >> -Alex >> >> -- >> You received this message because you are subscribed to the Google >> Groups "Clojure" group. >> To post to this group, send email to clo...@googlegroups.com >> Note that posts from new members are moderated - please be patient with >> your first post. >> To unsubscribe from this group, send email to >> clojure+u...@googlegroups.com >> For more options, visit this group at >> http://groups.google.com/group/clojure?hl=en >> --- >> You received this message because you are subscribed to the Google Groups >> "Clojure" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to clojure+u...@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> >> >> -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
