Oh.. and, I guess, the quick workaround for something like this would be to
have your Model be defined like so:
class myModel(db.Model):
myBigProp = db.BlobProperty()
and then you would have myBigProp be a compressed Model like so:
class myBigPropClass(db.Model):
prop1 = db.prop()
prop2 = db.prop()
...
prop100 = db.prop()
so.. when you constructed a myModel entity you did something like this
(naturally, this is pseudo code):
myProps = myBigPropClass(prop1 = blah, ..., prop100 = blah100)
compress = zlib.compress(db.model_to_protobuf(myProps).Encode(),9)
model = myModel(myBigProp = compress)
model.put()
I'm not sure if I can directly compress a protobuf... but that would be the
general idea.. mainly, the cost you spend on compression, decompression
isn't near as much as the cost you take on puts on the non-compressed data
or on the transfer during the get possibly..
Naturally, I would need to test this performance.. I know the put() cost of
the compressed data is much better in total.. I just haven't checked what
sort of hit you take on gets.
But, if you have really big entities.. you benefit greatly from the
compression. (Since the datastore is somewhere else.. and not right there
with your app instance)
Anyway, in the IO talk linked to above, Mr. Slatkin mentions that they are
looking into implementing a way to get specific properties with a get() ..
so.. it was on the radar.
On Wed, Mar 31, 2010 at 10:47 PM, Eli Jones <[email protected]> wrote:
> First off, that is a good talk. And I like it very much. But, it is still
> speaking of the workaround one can do to create a multi-part Model to query
> some part by __key__ and then grab the specific parent that contains the
> only props you want. You are still limited to partitioning your Models out
> into these specific parent, child relationships.. and then keeping track of
> those relationships and issuing queries based on said relationships.
>
> The particular case I'm talking about is one where you have a Model with
> 100 properties that are all unindexed... so.. you only have an index on the
> key_name.
>
> You can define multiple Models that are related.. but you will still have
> to deal with an index on the key_name for each Model (I am assuming).
>
> I want to know if it would ever be possible to have a single Model.. with a
> single key_name with one index.. and then be able to selectively pull
> particular properties for that Model..
>
> so.. if I want to update any three random properties, I issue a query that
> requests those properties.. updates them.. and puts() the entity back to the
> datastore.. only having to update those specific properties. In that
> situation, you wouldn't have the overhead of getting and putting properties
> that weren't going to be touched.
>
> Primarily, I'm wondering if there will ever be a way to avoid the overhead
> of pulling all properties for a Model when I only want to update a few of
> them. (The bandwidth used significantly impacts latency on the get() and
> the put() if the entities are fairly large)
>
>
> On Wed, Mar 31, 2010 at 10:09 PM, Raymond Ling <[email protected]>wrote:
>
>> I recommend this
>> video<http://www.youtube.com/watch?v=AgaL6NGpkB8&playnext_from=TL&videos=geh1T1kPh70>
>> (Youtube
>> Link:
>> http://www.youtube.com/watch?v=AgaL6NGpkB8&playnext_from=TL&videos=geh1T1kPh70
>> ).
>>
>> The simple way to avoid loading big property one time is to design the
>> property as
>> owned-relationship<http://code.google.com/appengine/docs/java/datastore/relationships.html>
>> .
>>
>> To keep the consistency between the object in your memory (maybe use
>> memcache) and entity in datastore, it's better to load all properties one
>> time (except for big properties that may cause performance issue).
>>
>> On Thu, Apr 1, 2010 at 8:52 AM, Eli Jones <[email protected]> wrote:
>>
>>> it would be nice to get only a few properties for an entity.. and then
>>> update those entities and the .put() would only need to send the updated
>>> values over the wire
>>
>>
>>
>>
>> --
>> Best Regards,
>> From Raymond Ling
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine" group.
>> To post to this group, send email to [email protected].
>> To unsubscribe from this group, send email to
>> [email protected]<google-appengine%[email protected]>
>> .
>> For more options, visit this group at
>> http://groups.google.com/group/google-appengine?hl=en.
>>
>
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
