Not being very familiar with the namespace apis, is it possible with the approach below to convert only a portion of the datastore object model and not duplicate the entire datastore?
For example - convert Users and Passwords but leave the weakly related Accounts in place. On Thu, Sep 23, 2010 at 8:27 AM, Eli Jones <[email protected]> wrote: > Yes, you'll need to duplicate your data. > > This is the cleanest way to do this.. in my mind. You would have > two separate versions of your app.. and you could easily switch back to your > old version of you realized some horrible mistake was made after switching > to the new version. > > The main work you would need to do for the changeover from Old Version to > New Version would be: > > 1. Write code that converts old datastore data to the New Version > namespace data. > 2. Test New Version datastore with converted data on New App version. > 3. Schedule a downtime late at night where the app is brought offline > (just upload a dummy app.yaml for the Old Version that points /.* to some > message page.. or you could be official and have a "Maintenance Period" > version of your app that you make live.. so that you make no changes > whatsoever to the "Old Version".), and the datastore conversion code is run > after you are comfortable that no users are using the site and are all > viewing the "maintenance period" messages. > 4. Once complete, make New Version app the live version. > > To me, it seems like a massive headache to try to convert the data in > place.. in the same datastore.. you have to have new model names or at least > some new property for "version" on your models.. and then your key_names > have to be different.. etc.. If you screw something up, that's a potential > headache for your live app.. with the New Version namespace.. you can > freely move ahead with experimenting with your coding.. and even make plenty > of changes to your models (if you think they are beneficial) without > thinking "I don't know.. it's going to be a pain in the ass to figure out > how to keep track of the old version and new version entities and now I have > to add a new model or property definition to my schema code.." > > With a namespace, you just have to add the correct mapping, conversion code > to your datastore converter code, and make schema changes to your New > Version app code.. and just leave the Old Version datastore and code alone. > > You can also get fancy with your converter.. (depending on how the > conversion actually works) and you can have your converter code get a cursor > with keys_only for each model, serialize that cursor and pass it off to a > task that then iterates through the cursor and breaks up the keys into > batches to be converted and fires off tasks to do the batches in parallel. > And, to make the process simpler, you could just used the deferred api.. > (so you don't have to deal with setting up task handler urls or anything > like that). > > However big your datastore is.. and however much money it might cost to > have your data duplicated (it costs $0.01 a day for 2 GB of data above the > free limit).. I think it would be worth it.. it would be valuable to learn > to do it this way, since a large scale site would (or should) do something > more like this. > > On Wed, Sep 22, 2010 at 9:39 PM, Kyle Baley <[email protected]> wrote: > >> That's an interesting idea. But if you do that, wouldn't you need to >> first copy all the data from the v1 namespace to the v2 namespace? >> >> On Sep 22, 7:07 pm, Eli Jones <[email protected]> wrote: >> > It might be useful for you to use a namespace for the new version of the >> > datastore. >> > >> > Thus, you could have the "new version" of the app deployed as a non-live >> > version of the app.. and code that "new version" to use the "new version >> > datastore" namespace. >> > >> > Then, when you are ready.. just change the live version of your app to >> the >> > "new version". >> > >> > Here's a link: >> > >> > http://code.google.com/appengine/docs/python/multitenancy/multitenanc. >> .. >> > >> > < >> http://code.google.com/appengine/docs/python/multitenancy/multitenanc.. >> .>So.. >> > you could just think of your version 1 datastore as "that old customer >> who >> > we're going to dump just as soon as our new version 2 datastore customer >> is >> > ready"... or something like that. >> > >> > It's better (I think) than adding a "version" property to all of your >> models >> > or trying to maintain model consistency between app versions. >> > >> > >> > >> > On Mon, Sep 20, 2010 at 2:29 PM, Kyle Baley <[email protected]> wrote: >> > > We've just released the first version of our application and are now >> > > looking at a problem we've been avoiding until now. Namely, what is >> > > the best way to upgrade the application to a new version that requires >> > > changes to the datastore. We're looking at two options: >> > >> > > 1) Big Bang Upgrade >> > > We take the application down and run an upgrade process to update all >> > > entities from version 1 to version 2. >> > >> > > Pros: Easy to maintain; intuitive >> > > Cons: App has to be taken down for a period of time, which will >> > > increase as time passes and more data is added to the datastore >> > > (potentially hitting the limit for long-running processes eventually) >> > > Question: What's a good way to take the app offline? >> > >> > > 2) Version Entities Individually >> > > Each entity has a version number and we have a series of commands, >> > > each one responsible for upgrading an entity from one version to the >> > > next. As we request entities, we check to see if it's the latest >> > > version. If not, we run each necessary upgrade command in sequence >> > > until it is the latest version. >> > >> > > Pros: No need to take the app offline; provides flexibility on whether >> > > to upgrade everything at once or piecemeal >> > > Cons: Not as intuitive; entities with different versions in the >> > > datastore (if that matters) >> > >> > > What do other people do to upgrade their datastore for a live >> > > application? >> > >> > > -- >> > > You received this message because you are subscribed to the Google >> Groups >> > > "Google App Engine" group. >> > > To post to this group, send email to >> [email protected]. >> > > To unsubscribe from this group, send email to >> > > [email protected]<google-appengine%[email protected]><google-appengine%2Bunsubscrib >> [email protected]> >> > > . >> > > For more options, visit this group at >> > >http://groups.google.com/group/google-appengine?hl=en. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]<google-appengine%[email protected]> >> . >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> >> > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]<google-appengine%[email protected]> > . > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
