On Wed, Jun 19, 2013 at 1:26 PM, Michael MacFadden <michael.macfad...@gmail.com> wrote: >>To start, I want to build a generic P2P OT container. This is a simple >>wrapper that contains a set of OT documents and defines a network >>protocol for keeping them in sync. The container needs to be able to >>talk to another instance of itself running somewhere else and >>syncronize documents between the two instances. >> >>Thats all I want this container to do - it should be as lightweight as >>possible, so we can port it to lots of different languages and >>environments. I want that code running in websites, in giant server >>farms, in vim, and everywhere in between. It won't have any database >>code, network code, users or a user interface (though it'll need APIs >>to support all of that stuff). At its core it just does OT + protocol >>work to syncronize documents. > > I strongly suggest we separate the OT Algorithms, the application level > protocol, and the network transport layer.
The network transport layer should definitely be separated out. I'd also like to separate out the OT functions themselves (a la share/ottypes). I'm imagining coupling the concurrency control system and the application level protocol. This coupling might not be needed - I don't know enough yet to be able to tell. I think we should focus on figuring out what OT system(s) to use first. >>The OT itself I imagine building along the lines of Torben Weis's P2P >>OT theory that he made in Lightwave: >>https://code.google.com/p/lightwave/ . Briefly, every operation gets a >>hash (like git). We add tombstones to wave's OT type and remove >>invertability, so the transform function supports TP2. We also add a >>prune function (inverse transform) which allows the history list to be >>reordered (so you don't have to transform out on every site). The hard >>part is figuring out which operations to sync, and which operations >>need to be reordered. I'd like to go over the details with Michael >>MacFadden and anyone else who's interested - there may well be a >>better system that we should use instead. If there is, I'd like to >>know about it now. > > I think this is another interesting area. One thing to consider is that I > am not sure if the linear model that wave used is even the right option. > If we start manipulating things like JSON Objects, Java Object, XML, or > nested documents, a hierarchical path mechanism may be best. Yes. For ShareJS's JSON OT type, operations are defined by a path (a list) and an operation at that path. I'm not convinced by this design anymore - we should definitely discuss it. > My other > instinct here is, we should make sure that we base the OT on something > that has been proven to be correct. There has been some work to evaluate > OT systems and prove that they are correct and solve the proper OT Puzzles > and support the required properties. > > Two other things that we need to consider. Beyond TP1 and TP2, there are > also IP1, IP2, and IP3 that you need to think about if you are going to > support group undo, which in my opinion wave needs to support. If you > can't undo things in collaboration, then you have problems. Using tombstones, I don't know how you can implement IP3. I've a mind to abandon formal invertibility, and simply rely on undo stacks for user level actions. But if we can find an architecture which meets our needs and can support invertibility, then that would be even better. > Basically, I am not confident that we know that wave's OT or what is in > lightwave is really what we want. I am not saying that they are not, I am > just saying that I don't think we really have stated what we NEED from the > OT and then proved that a particular approach salsifies those needs. > Simply having a demo of something that works in a toy environment is > likely to have holes that we don't see until much, much later in the > development cycle. > > My recommendation here would be for us to form a small committee to do a > literature review on the topic and to foster some technical discussion. > The result should be something in the wiki that lays out a plan that the > community can comment on. I completely agree. I don't know enough about what our other choices are. You're much better read than I am and you're better connected to the academic OT community. Would you be interested in organising / chairing this committee? >>I'm also thinking about full end-to-end encryption. Especially in the >>wake of the PRISM stuff, I'd quite like to make something secure. >>Snowden: "Encryption works. Properly implemented strong crypto systems >>are one of the few things that you can rely on. Unfortunately, >>endpoint security is so terrifically weak that NSA can frequently find >>ways around it." -- >>http://www.guardian.co.uk/world/2013/jun/17/edward-snowden-nsa-files-whist >>leblower >>. > I like this idea. There is a paper from France that talks about nested > hashing, digital signing, etc that proves the authenticity of operations > that might also be worth looking at. I will try to dig up the reference. Great! -Joseph
