:)
Actually, the decision about what documents (wavelets) *are* will
effect naming, which will in turn have ramifications throughout the
code.
Making a hybrid system is one of our options, but I think we still
need to pick a system.
-J
On Wed, Jun 19, 2013 at 2:19 PM, John Blossom <jblos...@gmail.com> wrote:
> Joseph,
>
> This is exactly the sort of transformational architecture that I have been
> hoping that you and others would pursue. Being fundamentally P2P but
> readily adaptable for scalable client-server architecture seems to fit
> today's mobile Web well. The three options that you outline might not wind
> up being either-or - perhaps different environments offer one or more
> handshake options.
>
> I'd love to sit in on this thread and glad to participate as needed and to
> work out documents etc.
>
> Thanks,
>
> Johm
> On Jun 19, 2013 2:23 PM, "Joseph Gentle" <jose...@gmail.com> wrote:
>
>> I've given half a dozen talks about ShareJS over the last 3 years, and
>> almost every time I give a talk, someone asks me whether you can use
>> ShareJS in a peer-to-peer way instead of just through a single server.
>>
>> "You say it works like subversion. Can it work like Git?"
>> "Can you have a document shared between multiple servers?"
>>
>> Sigh no. ShareJS & Wave's algorithms were invented in 1995. Back then,
>> it was news when someone put up a new website.
>>
>> "What about Wave Federation?" Appropriately, it works like IRC, but
>> using XML. Its complicated, its vulnerable to netsplits and its buggy.
>> I guess its like IRC except it doesn't actually work.
>>
>> So lets fix that! Lets modernize wave and make it federate properly.
>> On the way we have a great opportunity to make it simpler and cleaner.
>>
>>
>> To start, I want to build a generic P2P OT container. This is a simple
>> wrapper that contains a set of OT documents and defines a network
>> protocol for keeping them in sync. The container needs to be able to
>> talk to another instance of itself running somewhere else and
>> syncronize documents between the two instances.
>>
>> Thats all I want this container to do - it should be as lightweight as
>> possible, so we can port it to lots of different languages and
>> environments. I want that code running in websites, in giant server
>> farms, in vim, and everywhere in between. It won't have any database
>> code, network code, users or a user interface (though it'll need APIs
>> to support all of that stuff). At its core it just does OT + protocol
>> work to syncronize documents.
>>
>> What are the documents? Well, like ShareJS, I'd like to support
>> multiple different kinds of data. We'll need to be able to support
>> wave's conversation model, but I'd also like to support arbitrary
>> JSON. Doing OT over arbitrary JSON structures would allow other
>> applications to be built on top of wave, using wave as a data platform
>> ("Glorious messaging bus in the sky"). It'd also be super useful for
>> gadgets and user data.
>>
>> There's three models I can imagine for what wavelets could look like:
>>
>> Option 1: All documents in the container have a unique name and a
>> type. This is how ShareJS works. We could have a JSON type and a
>> wavelet type. This is simple, but not particularly extensible (it
>> makes it hard to embed JSON inside a conversation, and vice versa).
>>
>> Option 2: At the root of every document is a JSON object. Leaves in
>> the JSON structure can be subdocuments, which could be rich text for
>> blips, or any other type we think of down the road.
>>
>> Option 3: We make another layer, which can contain a set of documents.
>> So, a wavelet could contain a JSON document describing the
>> conversation structure, some rich text documents for blips and another
>> JSON document containing gadget data or something. Access control
>> rules are at the container level. This is (sort of) how wavelets work
>> today.
>>
>> The OT itself I imagine building along the lines of Torben Weis's P2P
>> OT theory that he made in Lightwave:
>> https://code.google.com/p/lightwave/ . Briefly, every operation gets a
>> hash (like git). We add tombstones to wave's OT type and remove
>> invertability, so the transform function supports TP2. We also add a
>> prune function (inverse transform) which allows the history list to be
>> reordered (so you don't have to transform out on every site). The hard
>> part is figuring out which operations to sync, and which operations
>> need to be reordered. I'd like to go over the details with Michael
>> MacFadden and anyone else who's interested - there may well be a
>> better system that we should use instead. If there is, I'd like to
>> know about it now.
>>
>>
>> Once thats built, we can start integrating it into WIAB. The simplest
>> way to do the client-server protocol and federation will be to simply
>> reuse the container's protocol (obviously wrapped for access control).
>> We could also strip it down for pure client-server interaction if we
>> want, to make it less chatty. (If we decide thats worthwhile.)
>>
>> I'm also thinking about full end-to-end encryption. Especially in the
>> wake of the PRISM stuff, I'd quite like to make something secure.
>> Snowden: "Encryption works. Properly implemented strong crypto systems
>> are one of the few things that you can rely on. Unfortunately,
>> endpoint security is so terrifically weak that NSA can frequently find
>> ways around it." --
>>
>> http://www.guardian.co.uk/world/2013/jun/17/edward-snowden-nsa-files-whistleblower
>> .
>>
>>
>> All of this should happen in the experiments branch (with a mirror on
>> github).
>>
>> The design decisions that we make here will be really hard to change
>> later, so I'd like to get this right. I'd like as much feedback as
>> possible. But please restrain yourself from complaining that its too
>> much work. You're not the boss of me :D
>>
>> Also, I expect the core OT piece to be no longer than a few thousand
>> lines. We can definitely pull that off - its just figuring out what
>> those lines are thats the tricky part.
>>
>> Cheers
>> Joseph
>>
