Ben, after seeing your draft patch I think we are taking different
approaches. First I am not going to use morphia. For example I think
DeltaStore doesn't represent an entity itself. Also I am not sure how many
collections this approach will generate (one collection per entity?) But
I don't have any experience in mapping libraries for MongoDB/noSQL so I
can't say much more.
About WaveletDeltaRecord serialization, this class contains:
- Applied delta, already serialized as a protobuffer. I'd store it as is.
- Transformed delta: it includes author, versions, timestamp, an a list of
WaveletOperation objects. I'd serialize this tree in mongoDB fields
So, at the end, in mongoDB, an unique document would be stored for this
delta, of course, including in the same document Wave and Wavele tIds. See
attached
MongoDbDeltaStore.java as a hint of my approach.
As far as I know, Memory and File Delta store issues are about scalabilty.
Except for WAVE-399 and WAVE-393, the file storage bug regarding the index
file for deltas.
BR,
Pablo
2014/1/3 Frank R. <renfeng...@gmail.com>
> I have one delta directory of 11MB at the moment, which consists of four
> files. One deltas file is 9.6MB.
>
>
> On Fri, Jan 3, 2014 at 12:45 AM, Ben Hegarty <heg...@gmail.com> wrote:
>
> > Does anyone know if the deltas or index files are likely to ever hit
> 16MB,
> > as that is the maximum document size inside mongodb and if we are then
> the
> > persistence will need to be pushed into the mongodb GridFS?
> >
> >
> > On Thu, Jan 2, 2014 at 9:58 AM, Frank R. <renfeng...@gmail.com> wrote:
> >
> > > This feature is on my wish list for Wave. It's a good news. Thank you~
> > >
> > >
> > > On Thu, Jan 2, 2014 at 5:24 PM, Ben Hegarty <heg...@gmail.com> wrote:
> > >
> > > > Hi Pablo,
> > > > Ironically I started work on this feature over the weekend as it is
> > next
> > > on
> > > > my list of must haves for my own personal project which also uses
> > > MongoDB.
> > > > I've had to upgrade the MongoDB driver and add in the morphia plugin
> > for
> > > > modelling the delta store which hopefully I'll be able to continue
> > > working
> > > > on over the weekend. My time is limited and so if you think you'll
> be
> > > able
> > > > to finish before me let me know and I'll try and help with any
> > knowledge
> > > > I've gained around mongoDB. It looks like getting the delta store
> > > finished
> > > > is going to be fiddly work from what I've seen so far, I started by
> > > > modelling the store on the memory version, but the first problem I've
> > > found
> > > > is that the '.' in mongoDB is a reserved character so I'm going to
> have
> > > to
> > > > start modelling the store using a mongoDB specific implementation.
> > > >
> > > > If it helps I've uploaded my current work
> > > > here<
> > > >
> > >
> >
> https://drive.google.com/file/d/0B5FF_Ld8SzsNQ2JONHdGU2xJUG8/edit?usp=sharing
> > > > >which
> > > > just keep in mind is very much just the beginning.
> > > >
> > > > HTH
> > > > Ben
> > > >
> > > >
> > > > On Thu, Jan 2, 2014 at 8:41 AM, Pablo Ojanguren <pablo...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I am a developer working for an EU research project. There, we are
> > > going
> > > > to
> > > > > use Wave (via Kune project) so we would like to help fixing bugs
> and
> > > > > improving the software along wiht the community. Currently I have
> > some
> > > > > available time for this for the next 10 months.
> > > > >
> > > > > After check out the issues, first thing I've paid attention is the
> > > > Delta's
> > > > > storage. I have reviewed server / persistence code and I think I
> can
> > > > start
> > > > > working in a mongoDB implementation.
> > > > >
> > > > > But maybe there is somebody else working on this. I would like to
> > join
> > > > them
> > > > > and to avoid duplicated efforts. Or if there is another outstanding
> > > > topics
> > > > > to suggest it would be great to know too.
> > > > >
> > > > > Thank you and Happy New Year.
> > > > >
> > > > > Pablo
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Mobile Phone: +447767 322 122
> > > > Home Phone: +4420 7737 6530
> > > >
> > >
> >
> >
> >
> > --
> > Regards
> > Ben
> >
>
package org.waveprotocol.box.server.persistence.mongodb;
import com.google.common.collect.ImmutableSet;
import com.mongodb.BasicDBObject;
import com.mongodb.DB;
import com.mongodb.DB.WriteConcern;
import com.mongodb.DBCollection;
import com.mongodb.DBCursor;
import com.mongodb.DBObject;
import com.mongodb.MongoException;
import com.mongodb.WriteResult;
import org.waveprotocol.box.common.ExceptionalIterator;
import org.waveprotocol.box.server.persistence.FileNotFoundPersistenceException;
import org.waveprotocol.box.server.persistence.PersistenceException;
import org.waveprotocol.box.server.waveserver.ByteStringMessage;
import org.waveprotocol.box.server.waveserver.DeltaStore;
import org.waveprotocol.box.server.waveserver.WaveletDeltaRecord;
import org.waveprotocol.wave.federation.Proto.ProtocolAppliedWaveletDelta;
import org.waveprotocol.wave.model.id.WaveId;
import org.waveprotocol.wave.model.id.WaveletId;
import org.waveprotocol.wave.model.id.WaveletName;
import org.waveprotocol.wave.model.operation.wave.TransformedWaveletDelta;
import org.waveprotocol.wave.model.version.HashedVersion;
import java.io.IOException;
import java.util.Collection;
import java.util.List;
public class MongoDbDeltaStore implements DeltaStore {
private static final String DELTAS_COLLECTION = "deltas";
private final DB database;
public MongoDbDeltaStore(DB database) {
this.database = database;
}
@Override
public DeltasAccess open(WaveletName waveletName) throws PersistenceException {
// TODO Consider using just one instance per wavelet or a singleton DeltaAccess
return new MongoDbDeltaCollection(waveletName);
}
@Override
public void delete(WaveletName waveletName) throws PersistenceException,
FileNotFoundPersistenceException {
// Build Criteria object
DBObject criteria = new BasicDBObject();
criteria.put("waveid", waveletName.waveId.serialise());
criteria.put("waveletid", waveletName.waveletId.serialise());
try {
// TODO upgrade MongoDB version to use new WriteConcern
WriteResult result = getDeltaDbCollection().remove(criteria, WriteConcern.NONE);
// TODO How to detect not found wavelets?
} catch (MongoException e) {
throw new PersistenceException(e);
}
}
@Override
public ImmutableSet<WaveletId> lookup(WaveId waveId) throws PersistenceException {
// Build Query object
DBObject query = new BasicDBObject();
query.put("waveid", waveId.serialise());
DBObject projection = new BasicDBObject();
projection.put("waveletid",1);
DBCursor cursor = null;
try {
// Perform query
cursor = getDeltaDbCollection().find(query,projection);
} catch (MongoException e) {
throw new PersistenceException(e);
}
// Fetch results
if (cursor == null || !cursor.hasNext()) {
return ImmutableSet.of();
} else {
ImmutableSet.Builder<WaveletId> builder = ImmutableSet.builder();
for (DBObject waveletIdDBObject : cursor) {
builder.add(WaveletId.deserialise((String) waveletIdDBObject.get("waveletid")));
}
return builder.build();
}
}
@Override
public ExceptionalIterator<WaveId, PersistenceException> getWaveIdIterator()
throws PersistenceException {
ImmutableSet.Builder<WaveId> builder = ImmutableSet.builder();
// TODO check if we can assume that wave deltas always contains info (non-empty waves)
// See MemoryDeltaStore.getWaveIdIterator()
try {
@SuppressWarnings("rawtypes")
List results = (List) getDeltaDbCollection().distinct("waveid");
for (Object o: results)
builder.add(WaveId.deserialise((String) o));
} catch (MongoException e) {
throw new PersistenceException(e);
}
return ExceptionalIterator.FromIterator.create(builder.build().iterator());
}
private DBCollection getDeltaDbCollection() {
return database.getCollection(DELTAS_COLLECTION);
}
}
