So here are my thoughts regarding this: if I have to map over all of my data re-writing it to prepare it for the tool to convert, why not just migrate to the other app while I'm doing that? Otherwise I'm going to pay to read and write it to fix it, then pay again to read it, transfer it, and rewrite it. This adds up fast if you've got much data.
Aside from the cost factor, the migration to HR is an opportunity to "fix" or improve your schemas. Things like long field names, unused indexes, entity groups, etc.... All of those items that require rewriting all of your data to adjust -- the migration is a great time to do it. On a migration I've recently been working on, I estimate I can reduce our storage use by about 50% with very minor changes (field names and indexes!). I've already migrated a couple apps, each time I've setup a job to map over my data, maybe do some conversions, put it in some suitable format (like csv or json) then send batches to the new app. On the new app side I write a simple 'receiver' handler that does any needed conversions and then writes the entities (in batches or one-by-one transactionally, depending on the case). Writing the sender and receiver handlers has usually been pretty quick for me, even with very complex data. The other advantage is that you may not need to move all of your data. Some items like stats that are computed from your original data, may be able to be recomputed by the exact same code as it comes in. Would love to hear other people's thoughts on this. Robert On Thu, Aug 4, 2011 at 21:28, Greg Darke <[email protected]> wrote: > Yes, this is the correct way to perform this operation. I would > suggest adding a new property to the model to store the keys and > update your code to use this new property. This will allow you to > continue using your application while you are performing your schema > migration. > > For example: > > # Old model definition > class MyModel(db.Model): > links = db.StringListProperty() > > # New model definition > class MyModel(db.Model): > links = db.StringListProperty() > link_keys = db.ListProperty(db.Key) # You may also want to set > indexed = False > > def apply_schema_change(self): > if not links: > # There is nothing to update > return > self.link_keys = [db.Key(str_key) for str_key in self.links] > self.links = [] # We don't use this anymore, so delete the data > > > Then you would call the apply_schema_change method after reading the > model in from datastore, ensuring that the data is now in the new > format. > > Then I would use the mapper framework to apply the new schema to all > of the entities in your datastore. > > > On 5 August 2011 10:53, Tim Hoffman <[email protected]> wrote: >> Hi John >> I have a db.ListProperty(db.String) or db.StringListProperty and I have >> stored keys in it, which of course will be in the str(key) form. >> The conversion tools doesn't convert this sort of thing mainly because >> unless there is an additional mechanism for >> describing what this field holds the conversion tools can't know. >> So you would need to something like >> Convert all the strings back to keys >> keys = [db.Key(i) for i in obj.some_list_property] >> then stick them in some property that is defined as db.ListProperty(db.Key) >> Now the trick will be working out the order of events and possibly >> performing multiple stages >> so you can get you data converted and make the sites work before and after >> transfer/conversion >> Haven't really worked out the order of events I have to go through. >> Rgds >> Tim >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To view this discussion on the web visit >> https://groups.google.com/d/msg/google-appengine/-/8_fgPFgpABAJ. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> > > > > -- > Greg Darke > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
