Hey Casey, One other thought, you could run over the data bundling and dumping it to the blobstore. Then pull the blobs to amazon. That might let you get a little more efficiency in the transfer. I've done it going in, should work just as well going out.
Aldo note that auto generated keys are not sequential nor strictly increasing. So you could potentially loose data. A solution I've used is to make a small adjustment to my models, I'll add an indexed 'batch' field that gets put on all new / updated entities. Do your main transfer using key order, then when your ready, grab everything with a batch value. Old data won't have a value for batch, so it won't be picked up in your final conversion. With a couple iterations you should be able to minimize your downtime. You could also use the remote API to fetch data during the final transfer stage. That should let you have zero downtime. Robert On Sep 1, 2011 8:22 AM, "Millisecond" <[email protected]> wrote: > > Won't rehash the pricing discussions, but because of the pricing > changes and the way it's been handled we're 90% sure we're going to > move to AWS. > > Anyways... now I'm tasked with figuring out how to get our almost TB > of data over there either into SimpleDB or into an RDS instance, not > sure yet. > > Have other people done the move, how did it go? Did you pull from > AWS / push from GAE? Take an intermediate backup and then load from > some other mechanism (thinking S3) into SDB / RDS? > > Our app is adding over 1 GB / day with 20MM reads and 20MM writes, and > we'd rather not take it offline for too long... Thinking of some > crazy scheme based on descending keys to move data over from key A > "downwards", shut off app, move over everything from key A "upwards" > as we're mostly only writing new data. Maybe special-case a few > object classes and use timestamps to detect deltas. Realize it's > pretty app-specific, but wondering if other people have tackled that > same problem and what their experiences were. > > Thanks for any and all help, > -Casey > > -- > You received this message because you are subscribed to the Google Groups "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to [email protected]. > For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
