Hey Casey, I think you can find some useful stuff in the SDK and maybe in the docs. I'm mobile now so I don't have links.
Robert On Sep 2, 2011 3:13 AM, "Casey Haakenson" <[email protected]> wrote: > > Thanks for the suggestions Robert. > > Is there documentation on using the remote API anywhere? A wrapper lib would be even better... > > Thanks, > -Casey > > On Fri, Sep 2, 2011 at 8:20 AM, Robert Kluin <[email protected]> wrote: >> >> Hey Casey, >> One other thought, you could run over the data bundling and dumping it to the blobstore. Then pull the blobs to amazon. That might let you get a little more efficiency in the transfer. I've done it going in, should work just as well going out. >> >> Aldo note that auto generated keys are not sequential nor strictly increasing. So you could potentially loose data. A solution I've used is to make a small adjustment to my models, I'll add an indexed 'batch' field that gets put on all new / updated entities. Do your main transfer using key order, then when your ready, grab everything with a batch value. Old data won't have a value for batch, so it won't be picked up in your final conversion. With a couple iterations you should be able to minimize your downtime. >> >> You could also use the remote API to fetch data during the final transfer stage. That should let you have zero downtime. >> >> Robert >> >> >> >> >> >> On Sep 1, 2011 8:22 AM, "Millisecond" <[email protected]> wrote: >> > >> > Won't rehash the pricing discussions, but because of the pricing >> > changes and the way it's been handled we're 90% sure we're going to >> > move to AWS. >> > >> > Anyways... now I'm tasked with figuring out how to get our almost TB >> > of data over there either into SimpleDB or into an RDS instance, not >> > sure yet. >> > >> > Have other people done the move, how did it go? Did you pull from >> > AWS / push from GAE? Take an intermediate backup and then load from >> > some other mechanism (thinking S3) into SDB / RDS? >> > >> > Our app is adding over 1 GB / day with 20MM reads and 20MM writes, and >> > we'd rather not take it offline for too long... Thinking of some >> > crazy scheme based on descending keys to move data over from key A >> > "downwards", shut off app, move over everything from key A "upwards" >> > as we're mostly only writing new data. Maybe special-case a few >> > object classes and use timestamps to detect deltas. Realize it's >> > pretty app-specific, but wondering if other people have tackled that >> > same problem and what their experiences were. >> > >> > Thanks for any and all help, >> > -Casey >> > >> > -- >> > You received this message because you are subscribed to the Google Groups "Google App Engine" group. >> > To post to this group, send email to [email protected]. >> > To unsubscribe from this group, send email to [email protected]. >> > For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. >> > >> >> -- >> You received this message because you are subscribed to the Google Groups "Google App Engine" group. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to [email protected]. >> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. > > > -- > You received this message because you are subscribed to the Google Groups "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to [email protected]. > For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
