Is that version number indexed? And if, how is performance on puts (and searches on that index)?
On Sep 28, 5:16 am, Robert Kluin <[email protected]> wrote: > I have several apps with complex models and relationships. Personally > I have found using a version property on my models has made most of my > upgrades and schema changes fairly straight forward. Although it does > sometimes take an interim version of the code capable of handling both > versions; or, possibly that sets the "version" explicitly on a put() > so that it can be upgraded later. > > In some cases I simply leave a version of the 'old' and 'new' logic in > place. When dealing with new data I use the new logic; when dealing > with existing data I use the version of the logic corresponding to the > entity or set of entities I am dealing with. Of course the structure > of my code itself facilitates this method, but it has proven very easy > to make even significant schema changes. Basically I push the interim > version then kick off my upgrade handlers. Works really well if you > do not want much, if any, downtime. YMMV. > > I like the idea of using namespaces to version the data too. But > figuring out how to either keep stuff in sync or going offline for a > full conversion could be tricky. I suppose you could add some type of > change indicator that gets set on _all_ of your models, then increment > it each time you run a conversion. That would let you identify what > has changed since your last run... possibly minimizing downtime? > > Robert > > > > On Mon, Sep 27, 2010 at 22:56, Eli Jones <[email protected]> wrote: > > Yes, when you did the official switch to put the new version of your app > > live for your customers.. you would need to take the app down and re-convert > > all the datastore data. > > Now, you could put a lot of time and energy figuring out a synchronization > > process.. but it seems like that would take a lot of extra coding. > > You'd have to have some process to compare the old datastore entities to the > > new datastore entities (either some sort of "datemodified" property or > > entity checksum.. or some sort of prop by prop comparison) and really.. > > seems like that would take more resources than just doing a straight > > re-conversion. > > I suppose if the re-conversion took hours and hours.. and cost a lot of > > money.. it might make sense. But then.. you'd have to figure.. would the > > re-conversion cost more money (and customer time) than the price of your > > time (and saved customer time) to code up a iterative sync process? > > Trying to straddle two databases (one for an old version of code.. and one > > for a new version of code) that stays synchronized can get pretty mind > > bending. But, that all depends on the complexity of your datastore.. > > So, synchronization is sort of a catch 22. > > If your datastore is so big and complex that synchronization takes less time > > than total re-conversion, the time to code out a sync process might be > > prohibitive. > > If your datastore is so simple that coding a synchronization process is > > easy, then re-conversion probably wouldn't take much time to run.. > > But, this is just a guess on my part.. I don't know what your datastore > > changes are going to look like. > > > On Mon, Sep 27, 2010 at 9:37 PM, Kyle Baley <[email protected]> wrote: > > >> Eli, > > >> I can see a benefit to this approach but isn't there a chance some > >> data would be missed? If customers are using the app while you're > >> testing the new version, wouldn't you need to re-import the data from > >> version 1 and convert it again while the app is down? > > >> On Sep 23, 11:27 am, Eli Jones <[email protected]> wrote: > >> > Yes, you'll need to duplicate your data. > > >> > This is the cleanest way to do this.. in my mind. You would have > >> > two separate versions of your app.. and you could easily switch back to > >> > your > >> > old version of you realized some horrible mistake was made after > >> > switching > >> > to the new version. > > >> > The main work you would need to do for the changeover from Old Version > >> > to > >> > New Version would be: > > >> > 1. Write code that converts old datastore data to the New Version > >> > namespace > >> > data. > >> > 2. Test New Version datastore with converted data on New App version. > >> > 3. Schedule a downtime late at night where the app is brought offline > >> > (just > >> > upload a dummy app.yaml for the Old Version that points /.* to some > >> > message > >> > page.. or you could be official and have a "Maintenance Period" version > >> > of > >> > your app that you make live.. so that you make no changes whatsoever to > >> > the > >> > "Old Version".), and the datastore conversion code is run after you are > >> > comfortable that no users are using the site and are all viewing the > >> > "maintenance period" messages. > >> > 4. Once complete, make New Version app the live version. > > >> > To me, it seems like a massive headache to try to convert the data in > >> > place.. in the same datastore.. you have to have new model names or at > >> > least > >> > some new property for "version" on your models.. and then your key_names > >> > have to be different.. etc.. If you screw something up, that's a > >> > potential > >> > headache for your live app.. with the New Version namespace.. you can > >> > freely move ahead with experimenting with your coding.. and even make > >> > plenty > >> > of changes to your models (if you think they are beneficial) without > >> > thinking "I don't know.. it's going to be a pain in the ass to figure > >> > out > >> > how to keep track of the old version and new version entities and now I > >> > have > >> > to add a new model or property definition to my schema code.." > > >> > With a namespace, you just have to add the correct mapping, conversion > >> > code > >> > to your datastore converter code, and make schema changes to your New > >> > Version app code.. and just leave the Old Version datastore and code > >> > alone. > > >> > You can also get fancy with your converter.. (depending on how the > >> > conversion actually works) and you can have your converter code get a > >> > cursor > >> > with keys_only for each model, serialize that cursor and pass it off to > >> > a > >> > task that then iterates through the cursor and breaks up the keys into > >> > batches to be converted and fires off tasks to do the batches in > >> > parallel. > >> > And, to make the process simpler, you could just used the deferred > >> > api.. > >> > (so you don't have to deal with setting up task handler urls or anything > >> > like that). > > >> > However big your datastore is.. and however much money it might cost to > >> > have > >> > your data duplicated (it costs $0.01 a day for 2 GB of data above the > >> > free > >> > limit).. I think it would be worth it.. it would be valuable to learn to > >> > do > >> > it this way, since a large scale site would (or should) do something > >> > more > >> > like this. > > >> > On Wed, Sep 22, 2010 at 9:39 PM, Kyle Baley <[email protected]> wrote: > >> > > That's an interesting idea. But if you do that, wouldn't you need to > >> > > first copy all the data from the v1 namespace to the v2 namespace? > > >> > > On Sep 22, 7:07 pm, Eli Jones <[email protected]> wrote: > >> > > > It might be useful for you to use a namespace for the new version of > >> > > > the > >> > > > datastore. > > >> > > > Thus, you could have the "new version" of the app deployed as a > >> > > > non-live > >> > > > version of the app.. and code that "new version" to use the "new > >> > > > version > >> > > > datastore" namespace. > > >> > > > Then, when you are ready.. just change the live version of your app > >> > > > to > >> > > the > >> > > > "new version". > > >> > > > Here's a link: > > >> > > > >http://code.google.com/appengine/docs/python/multitenancy/multitenanc... > > >> > > > <http://code.google.com/appengine/docs/python/multitenancy/multitenanc.. > >> > > .>So.. > >> > > > you could just think of your version 1 datastore as "that old > >> > > > customer > >> > > who > >> > > > we're going to dump just as soon as our new version 2 datastore > >> > > > customer > >> > > is > >> > > > ready"... or something like that. > > >> > > > It's better (I think) than adding a "version" property to all of > >> > > > your > >> > > models > >> > > > or trying to maintain model consistency between app versions. > > >> > > > On Mon, Sep 20, 2010 at 2:29 PM, Kyle Baley <[email protected]> wrote: > >> > > > > We've just released the first version of our application and are > >> > > > > now > >> > > > > looking at a problem we've been avoiding until now. Namely, what > >> > > > > is > >> > > > > the best way to upgrade the application to a new version that > >> > > > > requires > >> > > > > changes to the datastore. We're looking at two options: > > >> > > > > 1) Big Bang Upgrade > >> > > > > We take the application down and run an upgrade process to update > >> > > > > all > >> > > > > entities from version 1 to version 2. > > >> > > > > Pros: Easy to maintain; intuitive > >> > > > > Cons: App has to be taken down for a period of time, which will > >> > > > > increase as time passes and more data is added to the datastore > >> > > > > (potentially hitting the limit for long-running processes > >> > > > > eventually) > >> > > > > Question: What's a good way to take the app offline? > > >> > > > > 2) Version Entities Individually > >> > > > > Each entity has a version number and we have a series of commands, > >> > > > > each one responsible for upgrading an entity from one version to > >> > > > > the > >> > > > > next. As we request entities, we check to see if it's the latest > >> > > > > version. If not, we run each necessary upgrade command in sequence > >> > > > > until it is the latest version. > > >> > > > > Pros: No need to take the app offline; provides flexibility on > >> > > > > whether > >> > > > > to upgrade everything at once or piecemeal > >> > > > > Cons: Not as intuitive; entities with different versions in the > >> > > > > datastore (if that matters) > > >> > > > > What do other people do to upgrade their datastore for a live > >> > > > > application? > > >> > > > > -- > >> > > > > You received this message because you are subscribed to the Google > >> > > Groups > >> > > > > "Google App Engine" group. > > ... > > read more » -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
