Re: Delta store

2014-01-11 Thread Ben Hegarty
Just a note on the unit tests for the mongodb implementation, I noticed that they weren't actually using the test database and so the drop database and create db was having no effect, I fixed it and also fixed one or two of the tests that were relying on the db existing beforehand. so we should no

Re: Delta store

2014-01-09 Thread Pablo Ojanguren
Hi all, Ben, I've seen your patch for this and I had mine ready also, so I've sent it to review too. They both are pretty similar, so it should be good to open a discussion about to pros and cons of each approach. Best regards, Pablo 2014/1/3 Pablo Ojanguren > Ben, after seeing your draft pa

Re: Delta store

2014-01-03 Thread Pablo Ojanguren
Ben, after seeing your draft patch I think we are taking different approaches. First I am not going to use morphia. For example I think DeltaStore doesn't represent an entity itself. Also I am not sure how many collections this approach will generate (one collection per entity?) But I don't have

Re: Delta store

2014-01-02 Thread Frank R.
I have one delta directory of 11MB at the moment, which consists of four files. One deltas file is 9.6MB. On Fri, Jan 3, 2014 at 12:45 AM, Ben Hegarty wrote: > Does anyone know if the deltas or index files are likely to ever hit 16MB, > as that is the maximum document size inside mongodb and if

Re: Delta store

2014-01-02 Thread Ben Hegarty
Agreed, setting the WaveletDeltaRecord object as the top level document sounds sensible, I'll have a go at serializing it once I've built the rest of the model around it and created a workaround for the '.' issue that I'm seeing from attempting to save down the wave domain. I'll have to replace it

Re: Delta store

2014-01-02 Thread Pablo Ojanguren
The document size limit only matters in a document-per-wavelet data model approach. As long as updates (1) in existing documents are expensive operations in mongoDB a more granular approach would be suitable. I suggest a first simple data model with a document-per-delta approach, storing basicall

Re: Delta store

2014-01-02 Thread Ben Hegarty
Hey Yuri, I will do once I'm happy with the finished product, right now it's just a work in progress Regards Ben On Thursday, January 2, 2014, Yuri Z wrote: > Nice Ben. > Would you consider to create a code review with this patch on Review Board? > That will make it easier to review/comment on yo

Re: Delta store

2014-01-02 Thread Ben Hegarty
The question is more around how to go about modelling the data for mongodb, so that I don't hit any of Mongodb's limits. On Thursday, January 2, 2014, Ali Lown wrote: > I have seen the deltas files regularly hit >10MB. > The largest root conversation wavelet I have is 13300kB. (I think this > is

Re: Delta store

2014-01-02 Thread Yuri Z
Nice Ben. Would you consider to create a code review with this patch on Review Board? That will make it easier to review/comment on your patch. On Thu, Jan 2, 2014 at 7:07 PM, Pablo Ojanguren wrote: > Great! Ben, I am going to check out your patch and I will send you back > some feedback. By no

Re: Delta store

2014-01-02 Thread Pablo Ojanguren
Great! Ben, I am going to check out your patch and I will send you back some feedback. By now it's hard to know how much time the whole storage thing will take. And for sure I will need some help and knowledge from you. ;) 2014/1/2 Frank R. > This feature is on my wish list for Wave. It's a go

Re: Delta store

2014-01-02 Thread Ali Lown
I have seen the deltas files regularly hit >10MB. The largest root conversation wavelet I have is 13300kB. (I think this is only several hundred blips) My next question is why you are trying to store the serialized deltas for a whole wavelet in a single field in mongo? On 2 January 2014 16:45, Be

Re: Delta store

2014-01-02 Thread Ben Hegarty
Does anyone know if the deltas or index files are likely to ever hit 16MB, as that is the maximum document size inside mongodb and if we are then the persistence will need to be pushed into the mongodb GridFS? On Thu, Jan 2, 2014 at 9:58 AM, Frank R. wrote: > This feature is on my wish list for

Re: Delta store

2014-01-02 Thread Frank R.
This feature is on my wish list for Wave. It's a good news. Thank you~ On Thu, Jan 2, 2014 at 5:24 PM, Ben Hegarty wrote: > Hi Pablo, > Ironically I started work on this feature over the weekend as it is next on > my list of must haves for my own personal project which also uses MongoDB. > I've

Re: Delta store

2014-01-02 Thread Ben Hegarty
Hi Pablo, Ironically I started work on this feature over the weekend as it is next on my list of must haves for my own personal project which also uses MongoDB. I've had to upgrade the MongoDB driver and add in the morphia plugin for modelling the delta store which hopefully I'll be able to continu