Thanks for the suggestions Robert. Is there documentation on using the remote API anywhere? A wrapper lib would be even better...
Thanks, -Casey On Fri, Sep 2, 2011 at 8:20 AM, Robert Kluin <[email protected]> wrote: > Hey Casey, > One other thought, you could run over the data bundling and dumping it to > the blobstore. Then pull the blobs to amazon. That might let you get a > little more efficiency in the transfer. I've done it going in, should work > just as well going out. > > Aldo note that auto generated keys are not sequential nor strictly > increasing. So you could potentially loose data. A solution I've used is to > make a small adjustment to my models, I'll add an indexed 'batch' field > that gets put on all new / updated entities. Do your main transfer using > key order, then when your ready, grab everything with a batch value. Old > data won't have a value for batch, so it won't be picked up in your final > conversion. With a couple iterations you should be able to minimize your > downtime. > > You could also use the remote API to fetch data during the final transfer > stage. That should let you have zero downtime. > > Robert > > > > > > On Sep 1, 2011 8:22 AM, "Millisecond" <[email protected]> wrote: > > > > Won't rehash the pricing discussions, but because of the pricing > > changes and the way it's been handled we're 90% sure we're going to > > move to AWS. > > > > Anyways... now I'm tasked with figuring out how to get our almost TB > > of data over there either into SimpleDB or into an RDS instance, not > > sure yet. > > > > Have other people done the move, how did it go? Did you pull from > > AWS / push from GAE? Take an intermediate backup and then load from > > some other mechanism (thinking S3) into SDB / RDS? > > > > Our app is adding over 1 GB / day with 20MM reads and 20MM writes, and > > we'd rather not take it offline for too long... Thinking of some > > crazy scheme based on descending keys to move data over from key A > > "downwards", shut off app, move over everything from key A "upwards" > > as we're mostly only writing new data. Maybe special-case a few > > object classes and use timestamps to detect deltas. Realize it's > > pretty app-specific, but wondering if other people have tackled that > > same problem and what their experiences were. > > > > Thanks for any and all help, > > -Casey > > > > -- > > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > [email protected]. > > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
